Merge branch 'dev' into agent-master

* Let supervisor write more generally below /etc

* Let perl+plesk scripts run shells/write below etc

* Allow spaces after some cmdlines

* Add additional shell spawner.

* Add addl package mgmt binaries.

* Add addl cases for java + jenkins

Addl jar files to consider.

* Add addl jenkins-related cmdlines

Mostly related to node scripts run by jenkins

* Let python running some mesos tasks spawn shells

In this case marathon run by python

* Let ucf write below etc

Only below /etc/gconf for now.

* Let dpkg-reconfigur indirectly write below /etc

It may run programs that modify files below /etc

* Add files/dirs/prefixes for writes below root

Build a set of acceptable files/dirs/prefixes for writes below
/root. Mostly triggered by apps that run directly as root.

* Add addl shell spawn binaries.

* Also let java + sbt spawn shells in containers

Not seen only at host level

* Make sure the file below etc is /etc/

Make sure the file below /etc is really below the directory etc aka
/etc/xxx. Otherwise it would match a file /etcfoo.

* Let rancher healthcheck spawn shells

The name healthcheck is relatively innocuous so also look at the parent
process.

* Add addl shell container shell spawn binaries

* Add addl x2go binaries

* Let rabbitq write its config files

* Let rook write below /etc

toolbox.sh is fairly generic so add a condition based on the image name.

* Let consul-template spawn shells

* Add rook/toolbox as a trusted container

Their github pages recommend running privileged.

* Add addl mail binary that can setuid

* Let plesk autoinstaller spawn shells

The name autoinstaller is fairly generic so also look at the parent.

* Let php handlers write its config

* Let addl pkg-* binary write to /etc indirectly

* Add additional shell spawning binaries.

* Add ability to specify user trusted containers

New macro user_trusted_containers allows a user-provided set of
containers that are trusted and are allowed to run privileged.

* If npm runs node, let node spawn shells

* Let python run airflow via a shell.

* Add addl passenger commandlines (for shells)

* Add addl ways datadog can be run

* Let find run shells in containers.

* Add rpmq as a rpm binary

* Let httpd write below /etc/httpd/

* Let awstats/sa-update spawn shells

* Add container entrypoint as a shell

Some images have an extra shell level for image entrypoints.

* Add an additional jenkins commandline

* Let mysql write its config

* Let openvpn write its config

* Add addl root dirs/files

Also move /root/.java to be a general prefix.

* Let mysql_upgrade/opkg-cl spawn shells

* Allow login to perform dns lookups

With run with -h <host> to specify a remote host, some versions of login
will do a dns lookup to try to resolve the host.

* Let consul-template write haproxy config.

* Also let mysql indirectly edit its config

It might spawn a program to edit the config in addition to directly.

* Allow certain sed temp files below /etc/

* Allow debian binaries to indirectly write to /etc

They may spawn programs like sed, touch, etc to change files below /etc.

* Add additional root file

* Let rancher healthcheck be run more indirectly

The grandparent as well as parent of healthcheck can be tini.

* Add more cases for haproxy writing config

Allow more files as well as more scripts to update the config.

* Let vmtoolsd spawn shells on the host

* Add an additional innocuous entrypoint shell

* Let peer-finder (mongodb) spawn shells

* Split application rules to separate file.

Move the contents of application rules, which have never been enabled by
default, to a separate file. It's only installed in the mail falco packages.

* Add more build-related command lines

* Let perl running openresty spawn shells

* Let countly write nginx config

* Let confd spawn shells

* Also let aws spawn shells in containers.

.gitignore

@@ -1,6 +1,7 @@
 /build*
 *~
-test/falco_test.pyc
+*.pyc
+
 test/falco_tests.yaml
 test/traces-negative
 test/traces-positive

.travis.yml

@@ -21,18 +21,11 @@ install:
     - sudo python setup.py install
     - cd ../falco
 before_script:
-    - export KERNELDIR=/lib/modules/$(ls /lib/modules | sort | head -1)/build
+    - export KERNELDIR=/lib/modules/$(uname -r)/build
 script:
     - set -e
     - export CC="gcc-4.8"
     - export CXX="g++-4.8"
-    - wget https://s3.amazonaws.com/download.draios.com/dependencies/cmake-3.3.2.tar.gz
-    - tar -xzf cmake-3.3.2.tar.gz
-    - cd cmake-3.3.2
-    - ./bootstrap --prefix=/usr
-    - make
-    - sudo make install
-    - cd ..
     - mkdir build
     - cd build
     - cmake .. -DCMAKE_BUILD_TYPE=$BUILD_TYPE -DDRAIOS_DEBUG_FLAGS="-D_DEBUG -DNDEBUG"

CHANGELOG.md

@@ -2,9 +2,58 @@
 
 This file documents all notable changes to Falco. The release numbering uses [semantic versioning](http://semver.org).
 
+## v0.8.1
+
+Released 2017-10-10
+
+### Bug Fixes
+
+* Fix packaging to specify correct built-in config file [[#288](https://github.com/draios/falco/pull/288)]
+
+## v0.8.0
+
+Released 2017-10-10
+
+**Important**: the location for falco's configuration file has moved from `/etc/falco.yaml` to `/etc/falco/falco.yaml`. The default rules file has moved from `/etc/falco_rules.yaml` to `/etc/falco/falco_rules.yaml`. In addition, 0.8.0 has added a _local_ ruls file to `/etc/falco/falco_rules.local.yaml`. See [the documentation](https://github.com/draios/falco/wiki/Falco-Default-and-Local-Rules-Files) for more details.
+
+### Major Changes
+
+* Add the ability to append one list to another list by setting an `append: true` attribute. [[#264](https://github.com/draios/falco/pull/264)]
+* Add the ability to append one macro/rule to another list by setting an `append: true` attribute. [[#277](https://github.com/draios/falco/pull/277)]
+* Ensure that falco rules/config files are preserved across package upgrades/removes if modified. [[#278](https://github.com/draios/falco/pull/278)]
+* Add the notion of a "local" rules file that should contain modifications to the default falco rules file. [[#278](https://github.com/draios/falco/pull/278)]
+* When using json output, separately include the individual templated fields in the json object. [[#282](https://github.com/draios/falco/pull/282)]
+* Add the ability to keep a file/program pipe handle open across rule notifications. [[#283](https://github.com/draios/falco/pull/283)]
+* New argument `-V` validates rules file and immediately exits. [[#286](https://github.com/draios/falco/pull/286)]
+
+### Minor Changes
+
+* Minor updates to falco example programs [[#248](https://github.com/draios/falco/pull/248)] [[#275](https://github.com/draios/falco/pull/275)]
+* Also validate macros at rule parse time. [[#257](https://github.com/draios/falco/pull/257)]
+* Minor README typo fixes [[#276](https://github.com/draios/falco/pull/276)]
+* Add a government CLA (contributor license agreement). [[#263](https://github.com/draios/falco/pull/263)]
+* Add ability to only run rules with a priority >= some threshold [[#281](https://github.com/draios/falco/pull/281)]
+* Add ability to make output channels unbuffered [[#285](https://github.com/draios/falco/pull/285)]
+
+### Bug Fixes
+
+* Fix installation of falco on OSX [[#252](https://github.com/draios/falco/pull/252)]
+* Fix a bug that caused the trailing whitespace of a quoted string to be accidentally removed [[#254](https://github.com/draios/falco/pull/254)]
+* When multiple sets of kernel headers are installed, find the one for the running kernel [[#260](https://github.com/draios/falco/pull/260)]
+* Allow pathnames in rule/macro conditions to contain '.' characters [[#262](https://github.com/draios/falco/pull/262)]
+* Fix a bug where a list named "foo" would be substituted even if it were a substring of a longer word like "my_foo" [[#258](https://github.com/draios/falco/pull/258)]
+* Remove extra trailing newlines from rule output strings [[#265](https://github.com/draios/falco/pull/265)]
+* Improve build pathnames to avoid relative paths when possible [[#284](https://github.com/draios/falco/pull/284)]
+
+### Rule Changes
+
+* Significant changes to default ruleset to address FPs. These changes resulted from hundreds of hours of use in actual customer environments. [[#247](https://github.com/draios/falco/pull/247)] [[#259](https://github.com/draios/falco/pull/259)]
+* Add official gitlab EE docker image to list of known shell spawning images. Thanks @dkerwin! [[#270](https://github.com/draios/falco/pull/270)]
+* Add keepalived to list of shell spawning binaries. Thanks @dkerwin! [[#269](https://github.com/draios/falco/pull/269)]
+
 ## v0.7.0
 
-Released 2016-05-30
+Released 2017-05-30
 
 ### Major Changes
 
@@ -27,7 +76,7 @@ None.
 
 ## v0.6.1
 
-Released 2016-05-15
+Released 2017-05-15
 
 ### Major Changes
 
@@ -47,7 +96,7 @@ None
 
 ## v0.6.0
 
-Released 2016-03-29
+Released 2017-03-29
 
 ### Major Changes
 

CMakeLists.txt

@@ -7,7 +7,7 @@ if(NOT DEFINED FALCO_VERSION)
 endif()
 
 if(NOT DEFINED FALCO_ETC_DIR)
-	set(FALCO_ETC_DIR "/etc")
+	set(FALCO_ETC_DIR "/etc/falco")
 endif()
 
 if(NOT CMAKE_BUILD_TYPE)
@@ -43,7 +43,9 @@ set(PACKAGE_NAME "falco")
 set(PROBE_VERSION "${FALCO_VERSION}")
 set(PROBE_NAME "falco-probe")
 set(PROBE_DEVICE_NAME "falco")
-set(CMAKE_INSTALL_PREFIX /usr)
+if (CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT)
+	set(CMAKE_INSTALL_PREFIX /usr CACHE PATH "Default install path" FORCE)
+endif()
 
 set(CMD_MAKE make)
 
@@ -76,7 +78,7 @@ else()
         set(ZLIB_INCLUDE "${ZLIB_SRC}")
         set(ZLIB_LIB "${ZLIB_SRC}/libz.a")
         ExternalProject_Add(zlib
-		URL "http://download.draios.com/dependencies/zlib-1.2.8.tar.gz"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/zlib-1.2.8.tar.gz"
 		URL_MD5 "44d667c142d7cda120332623eab69f40"
 		CONFIGURE_COMMAND "./configure"
 		BUILD_COMMAND ${CMD_MAKE}
@@ -102,7 +104,7 @@ else()
         set(JQ_INCLUDE "${JQ_SRC}")
         set(JQ_LIB "${JQ_SRC}/.libs/libjq.a")
         ExternalProject_Add(jq
-                URL "http://download.draios.com/dependencies/jq-1.5.tar.gz"
+                URL "https://s3.amazonaws.com/download.draios.com/dependencies/jq-1.5.tar.gz"
                 URL_MD5 "0933532b086bd8b6a41c1b162b1731f9"
                 CONFIGURE_COMMAND ./configure --disable-maintainer-mode --enable-all-static --disable-dependency-tracking
                 BUILD_COMMAND ${CMD_MAKE} LDFLAGS=-all-static
@@ -132,7 +134,7 @@ else()
 	set(CURSES_LIBRARIES "${CURSES_BUNDLE_DIR}/lib/libncurses.a")
 	message(STATUS "Using bundled ncurses in '${CURSES_BUNDLE_DIR}'")
 	ExternalProject_Add(ncurses
-		URL "http://download.draios.com/dependencies/ncurses-6.0-20150725.tgz"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/ncurses-6.0-20150725.tgz"
 		URL_MD5 "32b8913312e738d707ae68da439ca1f4"
 		CONFIGURE_COMMAND ./configure --without-cxx --without-cxx-binding --without-ada --without-manpages --without-progs --without-tests --with-terminfo-dirs=/etc/terminfo:/lib/terminfo:/usr/share/terminfo
 		BUILD_COMMAND ${CMD_MAKE}
@@ -159,7 +161,7 @@ else()
 	set(B64_INCLUDE "${B64_SRC}/include")
 	set(B64_LIB "${B64_SRC}/src/libb64.a")
 	ExternalProject_Add(b64
-		URL "http://download.draios.com/dependencies/libb64-1.2.src.zip"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/libb64-1.2.src.zip"
 		URL_MD5 "a609809408327117e2c643bed91b76c5"
 		CONFIGURE_COMMAND ""
 		BUILD_COMMAND ${CMD_MAKE}
@@ -213,7 +215,7 @@ else()
 	message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'")
 
 	ExternalProject_Add(openssl
-		URL "http://download.draios.com/dependencies/openssl-1.0.2j.tar.gz"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/openssl-1.0.2j.tar.gz"
 		URL_MD5 "96322138f0b69e61b7212bc53d5e912b"
 		CONFIGURE_COMMAND ./config shared --prefix=${OPENSSL_INSTALL_DIR}
 		BUILD_COMMAND ${CMD_MAKE}
@@ -244,9 +246,9 @@ else()
 
 	ExternalProject_Add(curl
 		DEPENDS openssl
-		URL "http://download.draios.com/dependencies/curl-7.52.1.tar.bz2"
-		URL_MD5 "dd014df06ff1d12e173de86873f9f77a"
-		CONFIGURE_COMMAND ./configure ${CURL_SSL_OPTION} --disable-shared --enable-optimize --disable-curldebug --disable-rt --enable-http --disable-ftp --disable-file --disable-ldap --disable-ldaps --disable-rtsp --disable-telnet --disable-tftp --disable-pop3 --disable-imap --disable-smb --disable-smtp --disable-gopher --disable-sspi --disable-ntlm-wb --disable-tls-srp --without-winssl --without-darwinssl --without-polarssl --without-cyassl --without-nss --without-axtls --without-ca-path --without-ca-bundle --without-libmetalink --without-librtmp --without-winidn --without-libidn --without-nghttp2 --without-libssh2
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/curl-7.56.0.tar.bz2"
+		URL_MD5 "e0caf257103e0c77cee5be7e9ac66ca4"
+		CONFIGURE_COMMAND ./configure ${CURL_SSL_OPTION} --disable-shared --enable-optimize --disable-curldebug --disable-rt --enable-http --disable-ftp --disable-file --disable-ldap --disable-ldaps --disable-rtsp --disable-telnet --disable-tftp --disable-pop3 --disable-imap --disable-smb --disable-smtp --disable-gopher --disable-sspi --disable-ntlm-wb --disable-tls-srp --without-winssl --without-darwinssl --without-polarssl --without-cyassl --without-nss --without-axtls --without-ca-path --without-ca-bundle --without-libmetalink --without-librtmp --without-winidn --without-libidn --without-nghttp2 --without-libssh2 --disable-threaded-resolver
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
@@ -278,7 +280,7 @@ else()
 	set(LUAJIT_INCLUDE "${LUAJIT_SRC}")
 	set(LUAJIT_LIB "${LUAJIT_SRC}/libluajit.a")
 	ExternalProject_Add(luajit
-		URL "http://download.draios.com/dependencies/LuaJIT-2.0.3.tar.gz"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/LuaJIT-2.0.3.tar.gz"
 		URL_MD5 "f14e9104be513913810cd59c8c658dc0"
 		CONFIGURE_COMMAND ""
 		BUILD_COMMAND ${CMD_MAKE}
@@ -301,13 +303,14 @@ if(NOT USE_BUNDLED_LPEG)
 else()
 	set(LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg")
 	set(LPEG_LIB "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg/build/lpeg.a")
+	message(STATUS "Using bundled lpeg in '${LPEG_SRC}'")
 	set(LPEG_DEPENDENCIES "")
 	if(USE_BUNDLED_LUAJIT)
 		list(APPEND LPEG_DEPENDENCIES "luajit")
 	endif()
 	ExternalProject_Add(lpeg
 		DEPENDS ${LPEG_DEPENDENCIES}
-		URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz"
 		URL_MD5 "0aec64ccd13996202ad0c099e2877ece"
 		BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh" "${LPEG_SRC}/build"
 		BUILD_IN_SOURCE 1
@@ -340,8 +343,9 @@ else()
 
 	set(LIBYAML_SRC "${PROJECT_BINARY_DIR}/libyaml-prefix/src/libyaml/src")
 	set(LIBYAML_LIB "${LIBYAML_SRC}/.libs/libyaml.a")
+	message(STATUS "Using bundled libyaml in '${LIBYAML_SRC}'")
 	ExternalProject_Add(libyaml
-		URL "http://download.draios.com/dependencies/libyaml-0.1.4.tar.gz"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/libyaml-0.1.4.tar.gz"
 		URL_MD5 "4a4bced818da0b9ae7fc8ebc690792a7"
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
@@ -367,6 +371,7 @@ if(NOT USE_BUNDLED_LYAML)
 else()
 	set(LYAML_SRC "${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/ext/yaml")
 	set(LYAML_LIB "${LYAML_SRC}/.libs/yaml.a")
+	message(STATUS "Using bundled lyaml in '${LYAML_SRC}'")
 	set(LYAML_DEPENDENCIES "")
 	if(USE_BUNDLED_LUAJIT)
 		list(APPEND LYAML_DEPENDENCIES "luajit")
@@ -376,7 +381,7 @@ else()
 	endif()
 	ExternalProject_Add(lyaml
 		DEPENDS ${LYAML_DEPENDENCIES}
-		URL "http://download.draios.com/dependencies/lyaml-release-v6.0.tar.gz"
+		URL "https://s3.amazonaws.com/download.draios.com/dependencies/lyaml-release-v6.0.tar.gz"
                 URL_MD5 "dc3494689a0dce7cf44e7a99c72b1f30"
                 BUILD_COMMAND ${CMD_MAKE}
                 BUILD_IN_SOURCE 1
@@ -387,15 +392,19 @@ endif()
 install(FILES falco.yaml
 	DESTINATION "${FALCO_ETC_DIR}")
 
+add_subdirectory(rules)
+
 if(CMAKE_SYSTEM_NAME MATCHES "Linux")
 	add_subdirectory("${SYSDIG_DIR}/driver" "${PROJECT_BINARY_DIR}/driver")
 endif()
 add_subdirectory("${SYSDIG_DIR}/userspace/libscap" "${PROJECT_BINARY_DIR}/userspace/libscap")
 add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" "${PROJECT_BINARY_DIR}/userspace/libsinsp")
 
-add_subdirectory(scripts)
 set(FALCO_SINSP_LIBRARY sinsp)
-set(FALCO_SHARE_DIR ${CMAKE_INSTALL_PREFIX}/share/falco)
+set(FALCO_SHARE_DIR share/falco)
+set(FALCO_ABSOLUTE_SHARE_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}")
+set(FALCO_BIN_DIR bin)
+add_subdirectory(scripts)
 add_subdirectory(userspace/engine)
 add_subdirectory(userspace/falco)
 
@@ -416,7 +425,7 @@ set(CPACK_DEBIAN_PACKAGE_MAINTAINER "Sysdig <support@sysdig.com>")
 set(CPACK_DEBIAN_PACKAGE_SECTION "utils")
 set(CPACK_DEBIAN_PACKAGE_HOMEPAGE "http://www.sysdig.org")
 set(CPACK_DEBIAN_PACKAGE_DEPENDS "dkms (>= 2.1.0.0)")
-set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_BINARY_DIR}/scripts/debian/postinst;${CMAKE_BINARY_DIR}/scripts/debian/prerm;${PROJECT_SOURCE_DIR}/scripts/debian/postrm")
+set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_BINARY_DIR}/scripts/debian/postinst;${CMAKE_BINARY_DIR}/scripts/debian/prerm;${PROJECT_SOURCE_DIR}/scripts/debian/postrm;${PROJECT_SOURCE_DIR}/cpack/debian/conffiles")
 
 set(CPACK_RPM_PACKAGE_LICENSE "GPLv2")
 set(CPACK_RPM_PACKAGE_URL "http://www.sysdig.org")

README.md

@@ -2,7 +2,7 @@
 
 #### Latest release
 
-**v0.7.0**
+**v0.8.1**
 Read the [change log](https://github.com/draios/falco/blob/dev/CHANGELOG.md)
 
 Dev Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=dev)](https://travis-ci.org/draios/falco)<br />
@@ -33,10 +33,9 @@ Documentation
 
 Join the Community
 ---
-* Contact the [official mailing list](https://groups.google.com/forum/#!forum/falco) for support and to talk with other users.
 * Follow us on [Twitter](https://twitter.com/sysdig) for general falco and sysdig news.
 * This is our [blog](https://sysdig.com/blog/), where you can find the latest [falco](https://sysdig.com/blog/tag/falco/) posts.
-* Join our [Public Slack](https://sysdig.slack.com) channel for sysdig and falco announcements and discussions.
+* Join our [Public Slack](https://slack.sysdig.com) channel for sysdig and falco announcements and discussions.
 
 License Terms
 ---
@@ -61,15 +60,27 @@ As always, we are grateful for your past and present contributions to falco.
 
 **Individual contributions**: Individuals who wish to make contributions must review the [Individual Contributor License Agreement](./cla/falco_contributor_agreement.txt) and indicate agreement by adding the following line to every GIT commit message:
 
+```
 falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+```
 
 Use your real name; pseudonyms or anonymous contributions are not allowed.
 
 **Corporate contributions**: Employees of corporations, members of LLCs or LLPs, or others acting on behalf of a contributing entity, must review the [Corporate Contributor License Agreement](./cla/falco_corp_contributor_agreement.txt), must be an authorized representative of the contributing entity, and indicate agreement to it on behalf of the contributing entity by adding the following lines to every GIT commit message:
 
 ```
- falco-CLA-1.0-contributing-entity: Full Legal Name of Entity
- falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+falco-CLA-1.0-contributing-entity: Full Legal Name of Entity
+falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+```
+
+Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.
+
+**Government contributions**: Employees or officers of the United States Government, must review the [Government Contributor License Agreement](https://github.com/draios/falco/blob/dev/cla/falco_govt_contributor_agreement.txt), must be an authorized representative of the contributing entity, and indicate agreement to it on behalf of the contributing entity by adding the following lines to every GIT commit message:
+
+```
+falco-CLA-1.0-contributing-govt-entity: Full Legal Name of Entity
+falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+This file is a work of authorship of an employee or officer of the United States Government and is not subject to copyright in the United States under 17 USC 105.
 ```
 
 Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.

cla/falco_govt_contributor_agreement.txt

@@ -0,0 +1,33 @@
+DRAIOS, INC. <20> OPEN SOURCE CONTRIBUTION AGREEMENT FOR UNITED STATES GOVERNMENT CONTRIBUTING ENTITIES (<28>Agreement<6E>)
+
+Draios, Inc. (<28>Draios<6F> or <20>Sysdig<69>) welcomes the work of others on our open source software projects.  To contribute code back to our repositories, we require a contributing entity that is a United States Government agency to complete, and agree to, the Government Contributor Agreement (GCA) set forth here, by and through a designated authorized representative. This agreement clarifies the ability for us to use and incorporate the contributions of a government contributing entity in our projects and products. After agreeing to these terms, a contributing entity may contribute to our projects. To indicate the agreement of the contributing entity, an authorized representative shall follow the procedure set forth below under TO AGREE, after reading this Agreement. A <20>contributing entity<74> means any agency or unit of the United States government. We provide a separate CLA for individual contributors. 
+
+You accept and agree to the following terms and conditions for Your present and future Contributions that are submitted to Draios/Sysdig.
+
+1. Definitions.  "You" (or "Your") shall mean the contributing entity that has authored or otherwise has the right to contribute the Contribution, and that is making this Agreement with Draios/Sysdig. "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to Draios/Sysdig for inclusion in, or documentation of, any of the products owned or managed by Draios/Sysdig (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to Draios/Sysdig or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Draios/Sysdig for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
+
+2. Contributions Not Subject to Copyright. Each Contribution is a work authored by the United States Government or an employee or officer thereof and is not subject to copyright under 17 U.S.C. 105.
+
+3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Draios/Sysdig and to recipients of software distributed by Draios/Sysdig a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims that You have the right to license and that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity other than Draios/Sysdig institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
+
+4. You represent to Draios/Sysdig that You own or have the right to contribute Your Contributions to Draios/Sysdig, and that You are legally entitled to grant the license set forth above.
+
+5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which You are personally aware and which are associated with any part of Your Contributions. You represent that Your sign-off indicating assent to this Agreement includes the real name of a natural person who is an authorized representative of You, and not a pseudonym, and that You are not attempting or making an anonymous Contribution.
+
+6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions to Draios/Sysdig on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
+
+7. If You wish to submit work that is not Your original creation, You may submit it to Draios/Sysdig separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which You are aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
+
+8. You agree to notify Draios/Sysdig of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect.
+
+9. You understand and agree that this project and Your Contribution are public and that a record of the contribution, including all personal information that You submit with it, including the sign-off of Your authorized representative, may be stored by Draios/Sysdig indefinitely and may be redistributed to others. You understand and agree that Draios/Sysdig has no obligation to use any Contribution in any Draios/Sysdig project or product, and Draios/Sysdig may decline to accept Your Contributions or Draios/Sysdig may remove Your Contributions from Draios/Sysdig projects or products at any time without notice. You understand and agree that Draios/Sysdig is not and will not pay You any form of compensation, in currency, equity or otherwise, in exchange for Your Contributions or for Your assent to this Agreement. You understand and agree that You are independent of Draios/Sysdig and You are not, by entering into this Agreement or providing Your Contributions, becoming employed, hired as an independent contractor, or forming any other relationship with Draios/Sysdig relating to employment, compensation or ownership or involving any fiduciary obligation.
+
+TO AGREE:
+Add the following lines to every GIT commit message:
+
+falco-CLA-1.0-contributing-govt-entity: Full Legal Name of Entity
+falco-CLA-1.0-signed-off-by: Joe Smith joe.smith@email.com
+This file is a work of authorship of an employee or officer of the United States Government and is not subject to copyright in the United States under 17 USC 105.
+
+Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.
+

cpack/debian/conffiles

@@ -0,0 +1,4 @@
+/etc/falco/falco.yaml
+/etc/falco/falco_rules.yaml
+/etc/falco/application_rules.yaml
+/etc/falco/falco_rules.local.yaml

examples/nodejs-bad-rest-api/demo.yml

@@ -5,7 +5,7 @@ express_server:
   working_dir: /usr/src/app
   command: bash -c "npm install && node server.js"
   ports:
-    - "8080:8080"
+    - "8181:8181"
   volumes:
         - ${PWD}:/usr/src/app
 

examples/nodejs-bad-rest-api/server.js

@@ -2,19 +2,19 @@ var express    = require('express');        // call express
 var app        = express();                 // define our app using express
 var child_process = require('child_process');
 
-var port = process.env.PORT || 8080;        // set our port
+var port = process.env.PORT || 8181;        // set our port
 
 // ROUTES FOR OUR API
 // =============================================================================
 var router = express.Router();              // get an instance of the express Router
 
-// test route to make sure everything is working (accessed at GET http://localhost:8080/api)
+// test route to make sure everything is working (accessed at GET http://localhost:8181/api)
 router.get('/', function(req, res) {
     res.json({ message: 'API available'});
 });
 
 router.get('/exec/:cmd', function(req, res) {
-    var ret = child_process.spawnSync(req.params.cmd);
+    var ret = child_process.spawnSync(req.params.cmd, { shell: true});
     res.send(ret.stdout);
 });
 

falco.yaml

@@ -1,5 +1,15 @@
-# File containing Falco rules, loaded at startup.
-rules_file: /etc/falco_rules.yaml
+# File(s) containing Falco rules, loaded at startup.
+#
+# falco_rules.yaml ships with the falco package and is overridden with
+# every new software version. falco_rules.local.yaml is only created
+# if it doesn't exist. If you want to customize the set of rules, add
+# your customizations to falco_rules.local.yaml.
+#
+# The files will be read in the order presented here, so make sure if
+# you have overrides they appear in later files.
+rules_file:
+ - /etc/falco/falco_rules.yaml
+ - /etc/falco/falco_rules.local.yaml
 
 # Whether to output events in json or text
 json_output: false
@@ -15,6 +25,16 @@ log_syslog: true
 # "alert", "critical", "error", "warning", "notice", "info", "debug".
 log_level: info
 
+# Minimum rule priority level to load and run. All rules having a
+# priority more severe than this level will be loaded/run.  Can be one
+# of "emergency", "alert", "critical", "error", "warning", "notice",
+# "info", "debug".
+priority: debug
+
+# Whether or not output to any of the output channels below is
+# buffered. Defaults to true
+buffered_outputs: true
+
 # A throttling mechanism implemented as a token bucket limits the
 # rate of falco notifications. This throttling is controlled by the following configuration
 # options:
@@ -37,8 +57,13 @@ outputs:
 syslog_output:
   enabled: true
 
+# If keep_alive is set to true, the file will be opened once and
+# continuously written to, with each output message on its own
+# line. If keep_alive is set to false, the file will be re-opened
+# for each output message.
 file_output:
   enabled: false
+  keep_alive: false
   filename: ./events.txt
 
 stdout_output:
@@ -49,7 +74,15 @@ stdout_output:
 #         program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"
 #   - logging (alternate method than syslog):
 #         program: logger -t falco-test
+#   - send over a network connection:
+#         program: nc host.example.com 80
+
+# If keep_alive is set to true, the program will be started once and
+# continuously written to, with each output message on its own
+# line. If keep_alive is set to false, the program will be re-spawned
+# for each output message.
 
 program_output:
   enabled: false
+  keep_alive: false
   program: mail -s "Falco Notification" someone@example.com

rules/CMakeLists.txt

@@ -1,9 +1,11 @@
 if(NOT DEFINED FALCO_ETC_DIR)
-  set(FALCO_ETC_DIR "/etc")
+  set(FALCO_ETC_DIR "/etc/falco")
 endif()
 
 if(NOT DEFINED FALCO_RULES_DEST_FILENAME)
   set(FALCO_RULES_DEST_FILENAME "falco_rules.yaml")
+  set(FALCO_LOCAL_RULES_DEST_FILENAME "falco_rules.local.yaml")
+  set(FALCO_APP_RULES_DEST_FILENAME "application_rules.yaml")
 endif()
 
 if(DEFINED FALCO_COMPONENT)
@@ -11,9 +13,25 @@ install(FILES falco_rules.yaml
         COMPONENT "${FALCO_COMPONENT}"
 	DESTINATION "${FALCO_ETC_DIR}"
 	RENAME "${FALCO_RULES_DEST_FILENAME}")
+
+install(FILES falco_rules.local.yaml
+        COMPONENT "${FALCO_COMPONENT}"
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_LOCAL_RULES_DEST_FILENAME}")
+
+# Intentionally *not* installing application_rules.yaml. Not needed
+# when falco is embedded in other projects.
 else()
 install(FILES falco_rules.yaml
-        DESTINATION "${FALCO_ETC_DIR}"
-        RENAME "${FALCO_RULES_DEST_FILENAME}")
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_RULES_DEST_FILENAME}")
+
+install(FILES falco_rules.local.yaml
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_LOCAL_RULES_DEST_FILENAME}")
+
+install(FILES application_rules.yaml
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_APP_RULES_DEST_FILENAME}")
 endif()
 

rules/application_rules.yaml

@@ -0,0 +1,169 @@
+################################################################
+# By default all application-related rules are disabled for
+# performance reasons. Depending on the application(s) you use,
+# uncomment the corresponding rule definitions for
+# application-specific activity monitoring.
+################################################################
+
+# Elasticsearch ports
+- macro: elasticsearch_cluster_port
+  condition: fd.sport=9300
+- macro: elasticsearch_api_port
+  condition: fd.sport=9200
+- macro: elasticsearch_port
+  condition: elasticsearch_cluster_port or elasticsearch_api_port
+
+# - rule: Elasticsearch unexpected network inbound traffic
+#   desc: inbound network traffic to elasticsearch on a port other than the standard ports
+#   condition: user.name = elasticsearch and inbound and not elasticsearch_port
+#   output: "Inbound network traffic to Elasticsearch on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Elasticsearch unexpected network outbound traffic
+#   desc: outbound network traffic from elasticsearch on a port other than the standard ports
+#   condition: user.name = elasticsearch and outbound and not elasticsearch_cluster_port
+#   output: "Outbound network traffic from Elasticsearch on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+
+# ActiveMQ ports
+- macro: activemq_cluster_port
+  condition: fd.sport=61616
+- macro: activemq_web_port
+  condition: fd.sport=8161
+- macro: activemq_port
+  condition: activemq_web_port or activemq_cluster_port
+
+# - rule: Activemq unexpected network inbound traffic
+#   desc: inbound network traffic to activemq on a port other than the standard ports
+#   condition: user.name = activemq and inbound and not activemq_port
+#   output: "Inbound network traffic to ActiveMQ on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Activemq unexpected network outbound traffic
+#   desc: outbound network traffic from activemq on a port other than the standard ports
+#   condition: user.name = activemq and outbound and not activemq_cluster_port
+#   output: "Outbound network traffic from ActiveMQ on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+
+# Cassandra ports
+# https://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureFireWall_r.html
+- macro: cassandra_thrift_client_port
+  condition: fd.sport=9160
+- macro: cassandra_cql_port
+  condition: fd.sport=9042
+- macro: cassandra_cluster_port
+  condition: fd.sport=7000
+- macro: cassandra_ssl_cluster_port
+  condition: fd.sport=7001
+- macro: cassandra_jmx_port
+  condition: fd.sport=7199
+- macro: cassandra_port
+  condition: >
+    cassandra_thrift_client_port or
+    cassandra_cql_port or cassandra_cluster_port or
+    cassandra_ssl_cluster_port or cassandra_jmx_port
+
+# - rule: Cassandra unexpected network inbound traffic
+#   desc: inbound network traffic to cassandra on a port other than the standard ports
+#   condition: user.name = cassandra and inbound and not cassandra_port
+#   output: "Inbound network traffic to Cassandra on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Cassandra unexpected network outbound traffic
+#   desc: outbound network traffic from cassandra on a port other than the standard ports
+#   condition: user.name = cassandra and outbound and not (cassandra_ssl_cluster_port or cassandra_cluster_port)
+#   output: "Outbound network traffic from Cassandra on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Couchdb ports
+# https://github.com/davisp/couchdb/blob/master/etc/couchdb/local.ini
+- macro: couchdb_httpd_port
+  condition: fd.sport=5984
+- macro: couchdb_httpd_ssl_port
+  condition: fd.sport=6984
+# xxx can't tell what clustering ports are used. not writing rules for this
+# yet.
+
+# Fluentd ports
+- macro: fluentd_http_port
+  condition: fd.sport=9880
+- macro: fluentd_forward_port
+  condition: fd.sport=24224
+
+# - rule: Fluentd unexpected network inbound traffic
+#   desc: inbound network traffic to fluentd on a port other than the standard ports
+#   condition: user.name = td-agent and inbound and not (fluentd_forward_port or fluentd_http_port)
+#   output: "Inbound network traffic to Fluentd on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Tdagent unexpected network outbound traffic
+#   desc: outbound network traffic from fluentd on a port other than the standard ports
+#   condition: user.name = td-agent and outbound and not fluentd_forward_port
+#   output: "Outbound network traffic from Fluentd on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Gearman ports
+# http://gearman.org/protocol/
+# - rule: Gearman unexpected network outbound traffic
+#   desc: outbound network traffic from gearman on a port other than the standard ports
+#   condition: user.name = gearman and outbound and outbound and not fd.sport = 4730
+#   output: "Outbound network traffic from Gearman on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Zookeeper
+- macro: zookeeper_port
+  condition: fd.sport = 2181
+
+# Kafka ports
+# - rule: Kafka unexpected network inbound traffic
+#   desc: inbound network traffic to kafka on a port other than the standard ports
+#   condition: user.name = kafka and inbound and fd.sport != 9092
+#   output: "Inbound network traffic to Kafka on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# Memcached ports
+# - rule: Memcached unexpected network inbound traffic
+#   desc: inbound network traffic to memcached on a port other than the standard ports
+#   condition: user.name = memcached and inbound and fd.sport != 11211
+#   output: "Inbound network traffic to Memcached on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: Memcached unexpected network outbound traffic
+#   desc: any outbound network traffic from memcached. memcached never initiates outbound connections.
+#   condition: user.name = memcached and outbound
+#   output: "Unexpected Memcached outbound connection (connection=%fd.name)"
+#   priority: WARNING
+
+
+# MongoDB ports
+- macro: mongodb_server_port
+  condition: fd.sport = 27017
+- macro: mongodb_shardserver_port
+  condition: fd.sport = 27018
+- macro: mongodb_configserver_port
+  condition: fd.sport = 27019
+- macro: mongodb_webserver_port
+  condition: fd.sport = 28017
+
+# - rule: Mongodb unexpected network inbound traffic
+#   desc: inbound network traffic to mongodb on a port other than the standard ports
+#   condition: >
+#     user.name = mongodb and inbound and not (mongodb_server_port or
+#     mongodb_shardserver_port or mongodb_configserver_port or mongodb_webserver_port)
+#   output: "Inbound network traffic to MongoDB on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# MySQL ports
+# - rule: Mysql unexpected network inbound traffic
+#   desc: inbound network traffic to mysql on a port other than the standard ports
+#   condition: user.name = mysql and inbound and fd.sport != 3306
+#   output: "Inbound network traffic to MySQL on unexpected port (connection=%fd.name)"
+#   priority: WARNING
+
+# - rule: HTTP server unexpected network inbound traffic
+#   desc: inbound network traffic to a http server program on a port other than the standard ports
+#   condition: proc.name in (http_server_binaries) and inbound and fd.sport != 80 and fd.sport != 443
+#   output: "Inbound network traffic to HTTP Server on unexpected port (connection=%fd.name)"
+#   priority: WARNING

rules/falco_rules.local.yaml

@@ -0,0 +1,13 @@
+####################
+# Your custom rules!
+####################
+
+# Add new rules, like this one
+# - rule: The program "sudo" is run in a container
+#   desc: An event will trigger every time you run sudo in a container
+#   condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
+#   output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
+#   priority: ERROR
+#   tags: [users, container]
+
+# Or override/append to any rule, macro, or list from the Default Rules

scripts/CMakeLists.txt

@@ -7,6 +7,8 @@ file(COPY "${PROJECT_SOURCE_DIR}/scripts/debian/falco"
 file(COPY "${PROJECT_SOURCE_DIR}/scripts/rpm/falco"
 	DESTINATION "${PROJECT_BINARY_DIR}/scripts/rpm")
 
-install(PROGRAMS ${SYSDIG_DIR}/scripts/sysdig-probe-loader
-	DESTINATION bin
-	RENAME falco-probe-loader)
+if(CMAKE_SYSTEM_NAME MATCHES "Linux")
+	install(PROGRAMS ${SYSDIG_DIR}/scripts/sysdig-probe-loader
+		DESTINATION ${FALCO_BIN_DIR}
+		RENAME falco-probe-loader)
+endif()

test/falco_test.py

@@ -30,6 +30,7 @@ class FalcoTest(Test):
             self.trace_file = os.path.join(self.basedir, self.trace_file)
 
         self.json_output = self.params.get('json_output', '*', default=False)
+        self.priority = self.params.get('priority', '*', default='debug')
         self.rules_file = self.params.get('rules_file', '*', default=os.path.join(self.basedir, '../rules/falco_rules.yaml'))
 
         if not isinstance(self.rules_file, list):
@@ -295,7 +296,7 @@ class FalcoTest(Test):
             res = process.run("docker rm falco-test")
 
         elif self.package.endswith(".deb"):
-            cmdline = "dpkg -r falco"
+            cmdline = "dpkg --purge falco"
             self.log.debug("Uninstalling debian package via \"{}\"".format(cmdline))
             res = process.run(cmdline, timeout=120, sudo=True)
 
@@ -347,8 +348,8 @@ class FalcoTest(Test):
             trace_arg = "-e {}".format(self.trace_file)
 
         # Run falco
-        cmd = '{} {} {} -c {} {} -o json_output={} -v'.format(
-            self.falco_binary_path, self.rules_args, self.disabled_args, self.conf_file, trace_arg, self.json_output)
+        cmd = '{} {} {} -c {} {} -o json_output={} -o priority={} -v'.format(
+            self.falco_binary_path, self.rules_args, self.disabled_args, self.conf_file, trace_arg, self.json_output, self.priority)
 
         for tag in self.disable_tags:
             cmd += ' -T {}'.format(tag)

test/falco_tests.yaml

@@ -129,6 +129,21 @@ trace_files: !mux
       - rules/double_rule.yaml
     trace_file: trace_files/cat_write.scap
 
+  multiple_rules_suppress_info:
+    detect: True
+    detect_level:
+      - WARNING
+      - ERROR
+    priority: WARNING
+    detect_counts:
+      - open_from_cat: 8
+      - exec_from_cat: 1
+      - access_from_cat: 0
+    rules_file:
+      - rules/single_rule.yaml
+      - rules/double_rule.yaml
+    trace_file: trace_files/cat_write.scap
+
   multiple_rules_overriding:
     detect: False
     rules_file:
@@ -157,6 +172,68 @@ trace_files: !mux
       - rules/override_nested_list.yaml
     trace_file: trace_files/cat_write.scap
 
+  list_substring:
+    detect: False
+    rules_file:
+      - rules/list_substring.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_sub_front:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_sub_front.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_sub_mid:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_sub_mid.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_sub_end:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_sub_end.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_sub_bare:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_sub_bare.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_sub_whitespace:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_sub_whitespace.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_order:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_order.yaml
+    trace_file: trace_files/cat_write.scap
+
+  macro_order:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/macro_order.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_order:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/rule_order.yaml
+    trace_file: trace_files/cat_write.scap
+
   invalid_rule_output:
     exit_status: 1
     stderr_contains: "Runtime error: Error loading rules:.* Invalid output format 'An open was seen %not_a_real_field': 'invalid formatting token not_a_real_field'. Exiting."
@@ -517,3 +594,64 @@ trace_files: !mux
       - open_11: 1
       - open_12: 0
       - open_13: 0
+
+  list_append_failure:
+    exit_status: 1
+    stderr_contains: "List my_list has 'append' key but no list by that name already exists. Exiting"
+    rules_file:
+      - rules/list_append_failure.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_append:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/list_append.yaml
+    trace_file: trace_files/cat_write.scap
+
+  list_append_false:
+    detect: False
+    rules_file:
+      - rules/list_append_false.yaml
+    trace_file: trace_files/cat_write.scap
+
+  macro_append_failure:
+    exit_status: 1
+    stderr_contains: "Macro my_macro has 'append' key but no macro by that name already exists. Exiting"
+    rules_file:
+      - rules/macro_append_failure.yaml
+    trace_file: trace_files/cat_write.scap
+
+  macro_append:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/macro_append.yaml
+    trace_file: trace_files/cat_write.scap
+
+  macro_append_false:
+    detect: False
+    rules_file:
+      - rules/macro_append_false.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_append_failure:
+    exit_status: 1
+    stderr_contains: "Rule my_rule has 'append' key but no rule by that name already exists. Exiting"
+    rules_file:
+      - rules/rule_append_failure.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_append:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/rule_append.yaml
+    trace_file: trace_files/cat_write.scap
+
+  rule_append_false:
+    detect: False
+    rules_file:
+      - rules/rule_append_false.yaml
+    trace_file: trace_files/cat_write.scap
+

test/falco_traces.yaml.in

@@ -17,14 +17,14 @@ traces: !mux
     detect: True
     detect_level: INFO
     detect_counts:
-      - "File Open by Privileged Container": 19
+      - "Launch Privileged Container": 1
 
   container-sensitive-mount:
     trace_file: traces-positive/container-sensitive-mount.scap
     detect: True
     detect_level: INFO
     detect_counts:
-      - "Sensitive Mount by Container": 19
+      - "Launch Sensitive Mount Container": 1
 
   create-files-below-dev:
     trace_file: traces-positive/create-files-below-dev.scap
@@ -149,7 +149,7 @@ traces: !mux
   shell-in-container:
     trace_file: traces-positive/shell-in-container.scap
     detect: True
-    detect_level: NOTICE
+    detect_level: DEBUG
     detect_counts:
       - "Run shell in container": 1
 

test/rules/list_append.yaml

@@ -0,0 +1,12 @@
+- list: my_list
+  items: [not-cat]
+
+- list: my_list
+  append: true
+  items: [cat]
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and proc.name in (my_list)
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_append_failure.yaml

@@ -0,0 +1,3 @@
+- list: my_list
+  items: [not-cat]
+  append: true

test/rules/list_append_false.yaml

@@ -0,0 +1,12 @@
+- list: my_list
+  items: [cat]
+
+- list: my_list
+  append: false
+  items: [not-cat]
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and proc.name in (my_list)
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_order.yaml

@@ -0,0 +1,14 @@
+- list: cat_binaries
+  items: [not_cat]
+
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name in (cat_binaries)
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_sub_bare.yaml

@@ -0,0 +1,11 @@
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name=cat_binaries
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_sub_end.yaml

@@ -0,0 +1,11 @@
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name in (ls, cat_binaries)
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_sub_front.yaml

@@ -0,0 +1,11 @@
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name in (cat_binaries, ps)
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_sub_mid.yaml

@@ -0,0 +1,11 @@
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name in (ls, cat_binaries, ps)
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_sub_whitespace.yaml

@@ -0,0 +1,11 @@
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name= cat_binaries or proc.name=nopey
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/list_substring.yaml

@@ -0,0 +1,8 @@
+- list: my_list
+  items: ['"one string"']
+
+- rule: my_rule
+  desc: my description
+  condition: evt.type=open and fd.name in (file_my_list)
+  output: my output
+  priority: INFO

test/rules/macro_append.yaml

@@ -0,0 +1,12 @@
+- macro: my_macro
+  condition: proc.name=not-cat
+
+- macro: my_macro
+  append: true
+  condition: or proc.name=cat
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and my_macro
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/macro_append_failure.yaml

@@ -0,0 +1,3 @@
+- macro: my_macro
+  condition: proc.name=not-cat
+  append: true

test/rules/macro_append_false.yaml

@@ -0,0 +1,12 @@
+- macro: my_macro
+  condition: proc.name=cat
+
+- macro: my_macro
+  append: false
+  condition: proc.name=not-cat
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and my_macro
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/macro_order.yaml

@@ -0,0 +1,14 @@
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name in (not_cat)
+
+- macro: is_cat
+  condition: proc.name in (cat_binaries)
+  
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/rule_append.yaml

@@ -0,0 +1,9 @@
+- rule: my_rule
+  desc: A process named cat does an open
+  condition: evt.type=open and fd.name=not-a-real-file
+  output: "An open of /dev/null was seen (command=%proc.cmdline)"
+  priority: WARNING
+
+- rule: my_rule
+  append: true
+  condition: or fd.name=/dev/null

test/rules/rule_append_failure.yaml

@@ -0,0 +1,3 @@
+- rule: my_rule
+  condition: evt.type=open
+  append: true

test/rules/rule_append_false.yaml

@@ -0,0 +1,9 @@
+- rule: my_rule
+  desc: A process named cat does an open
+  condition: evt.type=open and fd.name=/dev/null
+  output: "An open of /dev/null was seen (command=%proc.cmdline)"
+  priority: WARNING
+
+- rule: my_rule
+  append: true
+  condition: and fd.name=not-a-real-file

test/rules/rule_order.yaml

@@ -0,0 +1,17 @@
+- list: cat_binaries
+  items: [cat]
+
+- macro: is_cat
+  condition: proc.name in (cat_binaries)
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and proc.name=not_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

userspace/engine/CMakeLists.txt

@@ -27,5 +27,3 @@ install(DIRECTORY lua
 	DESTINATION "${FALCO_SHARE_DIR}"
 	FILES_MATCHING PATTERN *.lua)
 endif()
-
-add_subdirectory("${PROJECT_SOURCE_DIR}/../falco/rules" "${PROJECT_BINARY_DIR}/rules")

userspace/engine/config_falco_engine.h.in

@@ -18,5 +18,5 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 
 #pragma once
 
-#define FALCO_ENGINE_LUA_DIR "${FALCO_SHARE_DIR}/lua/"
+#define FALCO_ENGINE_LUA_DIR "${FALCO_ABSOLUTE_SHARE_DIR}/lua/"
 #define FALCO_ENGINE_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/../falco/userspace/engine/lua/"

userspace/engine/falco_common.cpp

@@ -21,6 +21,16 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 #include "config_falco_engine.h"
 #include "falco_common.h"
 
+std::vector<std::string> falco_common::priority_names = {
+	"Emergency",
+	"Alert",
+	"Critical",
+	"Error",
+	"Warning",
+	"Notice",
+	"Informational",
+	"Debug"};
+
 falco_common::falco_common()
 {
 	m_ls = lua_open();

userspace/engine/falco_common.h

@@ -74,6 +74,22 @@ public:
 
 	void set_inspector(sinsp *inspector);
 
+        // Priority levels, as a vector of strings
+	static std::vector<std::string> priority_names;
+
+	// Same as numbers/indices into the above vector
+	enum priority_type
+	{
+		PRIORITY_EMERGENCY = 0,
+		PRIORITY_ALERT = 1,
+		PRIORITY_CRITICAL = 2,
+		PRIORITY_ERROR = 3,
+		PRIORITY_WARNING = 4,
+		PRIORITY_NOTICE = 5,
+		PRIORITY_INFORMATIONAL = 6,
+		PRIORITY_DEBUG = 7
+	};
+
 protected:
 	lua_State *m_ls;
 

userspace/engine/falco_engine.cpp

@@ -41,6 +41,7 @@ using namespace std;
 
 falco_engine::falco_engine(bool seed_rng)
 	: m_rules(NULL), m_next_ruleset_id(0),
+	  m_min_priority(falco_common::PRIORITY_DEBUG),
 	  m_sampling_ratio(1), m_sampling_multiplier(0),
 	  m_replace_container_info(false)
 {
@@ -89,7 +90,7 @@ void falco_engine::load_rules(const string &rules_content, bool verbose, bool al
 	bool json_output = false;
 	falco_formats::init(m_inspector, m_ls, json_output);
 
-	m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info);
+	m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info, m_min_priority);
 }
 
 void falco_engine::load_rules_file(const string &rules_filename, bool verbose, bool all_events)
@@ -134,6 +135,11 @@ void falco_engine::enable_rule_by_tag(const set<string> &tags, bool enabled)
 	enable_rule_by_tag(tags, enabled, m_default_ruleset);
 }
 
+void falco_engine::set_min_priority(falco_common::priority_type priority)
+{
+	m_min_priority = priority;
+}
+
 uint16_t falco_engine::find_ruleset_id(const std::string &ruleset)
 {
 	auto it = m_known_rulesets.lower_bound(ruleset);
@@ -148,6 +154,13 @@ uint16_t falco_engine::find_ruleset_id(const std::string &ruleset)
 	return it->second;
 }
 
+void falco_engine::evttypes_for_ruleset(std::vector<bool> &evttypes, const std::string &ruleset)
+{
+	uint16_t ruleset_id = find_ruleset_id(ruleset);
+
+	return m_evttype_filter->evttypes_for_ruleset(evttypes, ruleset_id);
+}
+
 unique_ptr<falco_engine::rule_result> falco_engine::process_event(sinsp_evt *ev, uint16_t ruleset_id)
 {
 	if(should_drop_evt())
@@ -178,7 +191,7 @@ unique_ptr<falco_engine::rule_result> falco_engine::process_event(sinsp_evt *ev,
 		res->evt = ev;
 		const char *p =  lua_tostring(m_ls, -3);
 		res->rule = p;
-		res->priority = lua_tostring(m_ls, -2);
+		res->priority_num = (falco_common::priority_type) lua_tonumber(m_ls, -2);
 		res->format = lua_tostring(m_ls, -1);
 		lua_pop(m_ls, 3);
 	}

userspace/engine/falco_engine.h

@@ -67,10 +67,13 @@ public:
 	// Wrapper that assumes the default ruleset
 	void enable_rule_by_tag(const std::set<std::string> &tags, bool enabled);
 
+	// Only load rules having this priority or more severe.
+	void set_min_priority(falco_common::priority_type priority);
+
 	struct rule_result {
 		sinsp_evt *evt;
 		std::string rule;
-		std::string priority;
+		falco_common::priority_type priority_num;
 		std::string format;
 	};
 
@@ -82,6 +85,12 @@ public:
 	//
 	uint16_t find_ruleset_id(const std::string &ruleset);
 
+	//
+	// Given a ruleset, fill in a bitset containing the event
+	// types for which this ruleset can run.
+	//
+	void evttypes_for_ruleset(std::vector<bool> &evttypes, const std::string &ruleset);
+
 	//
 	// Given an event, check it against the set of rules in the
 	// engine and if a matching rule is found, return details on
@@ -158,6 +167,7 @@ private:
 	uint16_t m_next_ruleset_id;
 	std::map<string, uint16_t> m_known_rulesets;
 	std::unique_ptr<sinsp_evttype_filter> m_evttype_filter;
+	falco_common::priority_type m_min_priority;
 
 	//
 	// Here's how the sampling ratio and multiplier influence

userspace/engine/formats.cpp

@@ -92,6 +92,7 @@ int falco_formats::free_formatters(lua_State *ls)
 int falco_formats::format_event (lua_State *ls)
 {
 	string line;
+	string json_line;
 
 	if (!lua_isstring(ls, -1) ||
 	    !lua_isstring(ls, -2) ||
@@ -109,6 +110,20 @@ int falco_formats::format_event (lua_State *ls)
 
 	try {
 		s_formatters->tostring(evt, sformat, &line);
+
+		if(s_json_output)
+		{
+			s_inspector->set_buffer_format(sinsp_evt::PF_JSON);
+			s_formatters->tostring(evt, sformat, &json_line);
+
+			// The formatted string might have a leading newline. If it does, remove it.
+			if (json_line[0] == '\n')
+			{
+				json_line.erase(0, 1);
+			}
+
+			s_inspector->set_buffer_format(sinsp_evt::PF_NORMAL);
+		}
 	}
 	catch (sinsp_exception& e)
 	{
@@ -117,13 +132,15 @@ int falco_formats::format_event (lua_State *ls)
 		lua_error(ls);
 	}
 
-	// For JSON output, the formatter returned just the output
-	// string containing the format text and values. Use this to
-	// build a more detailed object containing the event time,
-	// rule, severity, full output, and fields.
+	// For JSON output, the formatter returned a json-as-text
+	// object containing all the fields in the original format
+	// message as well as the event time in ns. Use this to build
+	// a more detailed object containing the event time, rule,
+	// severity, full output, and fields.
 	if (s_json_output) {
 		Json::Value event;
 		Json::FastWriter writer;
+		string full_line;
 
 		// Convert the time-as-nanoseconds to a more json-friendly ISO8601.
 		time_t evttime = evt->get_ts()/1000000000;
@@ -138,16 +155,26 @@ int falco_formats::format_event (lua_State *ls)
 		event["time"] = iso8601evttime;
 		event["rule"] = rule;
 		event["priority"] = level;
+		// This is the filled-in output line.
 		event["output"] = line;
 
-		line = writer.write(event);
+		full_line = writer.write(event);
 
 		// Json::FastWriter may add a trailing newline. If it
 		// does, remove it.
-		if (line[line.length()-1] == '\n')
+		if (full_line[full_line.length()-1] == '\n')
 		{
-			line.resize(line.length()-1);
+			full_line.resize(full_line.length()-1);
 		}
+
+		// Cheat-graft the output from the formatter into this
+		// string. Avoids an unnecessary json parse just to
+		// merge the formatted fields at the object level.
+		full_line.pop_back();
+		full_line.append(", \"output_fields\": ");
+		full_line.append(json_line);
+		full_line.append("}");
+		line = full_line;
 	}
 
 	lua_pushstring(ls, line.c_str());

userspace/engine/lua/compiler.lua

@@ -20,6 +20,7 @@ local compiler = {}
 
 compiler.verbose = false
 compiler.all_events = false
+compiler.trim = parser.trim
 
 function compiler.set_verbose(verbose)
    compiler.verbose = verbose
@@ -58,14 +59,15 @@ end
    definition uses another macro).
 
 --]]
-function expand_macros(ast, defs, changed)
 
-   function copy(obj)
-      if type(obj) ~= 'table' then return obj end
-      local res = {}
-      for k, v in pairs(obj) do res[copy(k)] = copy(v) end
-      return res
-   end
+function copy_ast_obj(obj)
+   if type(obj) ~= 'table' then return obj end
+   local res = {}
+   for k, v in pairs(obj) do res[copy_ast_obj(k)] = copy_ast_obj(v) end
+   return res
+end
+
+function expand_macros(ast, defs, changed)
 
    if (ast.type == "Rule") then
       return expand_macros(ast.filter, defs, changed)
@@ -74,7 +76,7 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.value.value] == nil) then
             error("Undefined macro '".. ast.value.value .. "' used in filter.")
          end
-         ast.value = copy(defs[ast.value.value])
+         ast.value = copy_ast_obj(defs[ast.value.value])
          changed = true
 	 return changed
       end
@@ -86,7 +88,7 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.left.value] == nil) then
             error("Undefined macro '".. ast.left.value .. "' used in filter.")
          end
-         ast.left = copy(defs[ast.left.value])
+         ast.left = copy_ast_obj(defs[ast.left.value])
          changed = true
       end
 
@@ -94,7 +96,7 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.right.value] == nil) then
             error("Undefined macro ".. ast.right.value .. " used in filter.")
          end
-         ast.right = copy(defs[ast.right.value])
+         ast.right = copy_ast_obj(defs[ast.right.value])
          changed = true
       end
 
@@ -107,7 +109,7 @@ function expand_macros(ast, defs, changed)
          if (defs[ast.argument.value] == nil) then
             error("Undefined macro ".. ast.argument.value .. " used in filter.")
          end
-         ast.argument = copy(defs[ast.argument.value])
+         ast.argument = copy_ast_obj(defs[ast.argument.value])
          changed = true
       end
       return expand_macros(ast.argument, defs, changed)
@@ -281,7 +283,7 @@ function get_evttypes(name, ast, source)
    return evttypes
 end
 
-function compiler.compile_macro(line, list_defs)
+function compiler.compile_macro(line, macro_defs, list_defs)
 
    for name, items in pairs(list_defs) do
       line = string.gsub(line, name, table.concat(items, ", "))
@@ -300,6 +302,21 @@ function compiler.compile_macro(line, list_defs)
       check_for_ignored_syscalls_events(ast, 'macro', line)
    end
 
+   -- Simply as a validation step, try to expand all macros in this
+   -- macro's condition. This changes the ast, so we make a copy
+   -- first.
+   local ast_copy = copy_ast_obj(ast)
+
+   if (ast.type == "Rule") then
+      -- Line is a filter, so expand macro references
+      repeat
+	 expanded  = expand_macros(ast_copy, macro_defs, false)
+      until expanded == false
+
+   else
+      error("Unexpected top-level AST type: "..ast.type)
+   end
+
    return ast
 end
 
@@ -309,7 +326,12 @@ end
 function compiler.compile_filter(name, source, macro_defs, list_defs)
 
    for name, items in pairs(list_defs) do
-      source = string.gsub(source, name, table.concat(items, ", "))
+      local begin_name_pat = "^("..name..")([%s(),=])"
+      local mid_name_pat = "([%s(),=])("..name..")([%s(),=])"
+      local end_name_pat = "([%s(),=])("..name..")$"
+      source = string.gsub(source, begin_name_pat, table.concat(items, ", ").."%2")
+      source = string.gsub(source, mid_name_pat, "%1"..table.concat(items, ", ").."%3")
+      source = string.gsub(source, end_name_pat, "%1"..table.concat(items, ", "))
    end
 
    local ast, error_msg = parser.parse_filter(source)

userspace/engine/lua/parser.lua

@@ -127,10 +127,11 @@ function trim(s)
    if (type(s) ~= "string") then return s end
   return (s:gsub("^%s*(.-)%s*$", "%1"))
 end
+parser.trim = trim
 
 local function terminal (tag)
    -- Rather than trim the whitespace in this way, it would be nicer to exclude it from the capture...
-   return token(V(tag), tag) / function (tok) return { type = tag, value = trim(tok)} end
+   return token(V(tag), tag) / function (tok) val = tok; if tag ~= "String" then val = trim(tok) end; return { type = tag, value = val} end
 end
 
 local function unaryboolop (op, e)
@@ -237,7 +238,7 @@ local G = {
   Identifier = V"idStart" * V"idRest"^0;
   Macro = V"idStart" * V"idRest"^0 * -P".";
   Int = digit^1;
-  PathString = (alnum + S'-_/*?')^1;
+  PathString = (alnum + S'.-_/*?')^1;
   Index = V"Int" + V"PathString";
   FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Index" * P"]")^-1;
   Name = C(V"Identifier") * -V"idRest";

userspace/engine/lua/rule_loader.lua

@@ -58,6 +58,17 @@ function map(f, arr)
    return res
 end
 
+priorities = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"}
+
+local function priority_num_for(s)
+   s = string.lower(s)
+   for i,v in ipairs(priorities) do
+      if (string.find(string.lower(v), "^"..s)) then
+	 return i - 1 -- (numbers start at 0, lua indices start at 1)
+      end
+   end
+   error("Invalid priority level: "..s)
+end
 
 --[[
    Take a filter AST and set it up in the libsinsp runtime, using the filter API.
@@ -171,7 +182,7 @@ function table.tostring( tbl )
 end
 
 
-function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replace_container_info)
+function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replace_container_info, min_priority)
 
    compiler.set_verbose(verbose)
    compiler.set_all_events(all_events)
@@ -208,7 +219,23 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	    end
 	 end
 
-	 state.macros_by_name[v['macro']] = v
+	 -- Possibly append to the condition field of an existing macro
+	 append = false
+
+	 if v['append'] then
+	    append = v['append']
+	 end
+
+	 if append then
+	    if state.macros_by_name[v['macro']] == nil then
+	       error ("Macro " ..v['macro'].. " has 'append' key but no macro by that name already exists")
+	    end
+
+	    state.macros_by_name[v['macro']]['condition'] = state.macros_by_name[v['macro']]['condition'] .. " " .. v['condition']
+
+	 else
+	    state.macros_by_name[v['macro']] = v
+	 end
 
       elseif (v['list']) then
 
@@ -222,7 +249,24 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	    end
 	 end
 
-	 state.lists_by_name[v['list']] = v
+	 -- Possibly append to an existing list
+	 append = false
+
+	 if v['append'] then
+	    append = v['append']
+	 end
+
+	 if append then
+	    if state.lists_by_name[v['list']] == nil then
+	       error ("List " ..v['list'].. " has 'append' key but no list by that name already exists")
+	    end
+
+	    for i, elem in ipairs(v['items']) do
+	       table.insert(state.lists_by_name[v['list']]['items'], elem)
+	    end
+	 else
+	    state.lists_by_name[v['list']] = v
+	 end
 
       elseif (v['rule']) then
 
@@ -230,21 +274,54 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 	    error ("Missing name in rule")
 	 end
 
-	 for i, field in ipairs({'condition', 'output', 'desc', 'priority'}) do
-	    if (v[field] == nil) then
-	       error ("Missing "..field.." in rule with name "..v['rule'])
+	 -- Possibly append to the condition field of an existing rule
+	 append = false
+
+	 if v['append'] then
+	    append = v['append']
+	 end
+
+	 if append then
+
+	    -- For append rules, all you need is the condition
+	    for i, field in ipairs({'condition'}) do
+	       if (v[field] == nil) then
+		  error ("Missing "..field.." in rule with name "..v['rule'])
+	       end
+	    end
+
+	    if state.rules_by_name[v['rule']] == nil then
+	       error ("Rule " ..v['rule'].. " has 'append' key but no rule by that name already exists")
+	    end
+
+	    state.rules_by_name[v['rule']]['condition'] = state.rules_by_name[v['rule']]['condition'] .. " " .. v['condition']
+
+	 else
+
+	    for i, field in ipairs({'condition', 'output', 'desc', 'priority'}) do
+	       if (v[field] == nil) then
+		  error ("Missing "..field.." in rule with name "..v['rule'])
+	       end
+	    end
+
+	    -- Convert the priority-as-string to a priority-as-number now
+	    v['priority_num'] = priority_num_for(v['priority'])
+
+	    if v['priority_num'] <= min_priority then
+	       -- Note that we can overwrite rules, but the rules are still
+	       -- loaded in the order in which they first appeared,
+	       -- potentially across multiple files.
+	       if state.rules_by_name[v['rule']] == nil then
+		  state.ordered_rule_names[#state.ordered_rule_names+1] = v['rule']
+	       end
+
+	       -- The output field might be a folded-style, which adds a
+	       -- newline to the end. Remove any trailing newlines.
+	       v['output'] = compiler.trim(v['output'])
+
+	       state.rules_by_name[v['rule']] = v
 	    end
 	 end
-
-	 -- Note that we can overwrite rules, but the rules are still
-	 -- loaded in the order in which they first appeared,
-	 -- potentially across multiple files.
-	 if state.rules_by_name[v['rule']] == nil then
-	    state.ordered_rule_names[#state.ordered_rule_names+1] = v['rule']
-	 end
-
-	 state.rules_by_name[v['rule']] = v
-
       else
 	 error ("Unknown rule object: "..table.tostring(v))
       end
@@ -283,7 +360,7 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac
 
       local v = state.macros_by_name[name]
 
-      local ast = compiler.compile_macro(v['condition'], state.lists)
+      local ast = compiler.compile_macro(v['condition'], state.macros, state.lists)
       state.macros[v['macro']] = ast.filter.value
    end
 
@@ -443,7 +520,7 @@ function on_event(evt_, rule_id)
    -- Prefix output with '*' so formatting is permissive
    output = "*"..rule.output
 
-   return rule.rule, rule.priority, output
+   return rule.rule, rule.priority_num, output
 end
 
 function print_stats()

userspace/engine/rules.cpp

@@ -145,7 +145,8 @@ void falco_rules::enable_rule(string &rule, bool enabled)
 
 void falco_rules::load_rules(const string &rules_content,
 			     bool verbose, bool all_events,
-			     string &extra, bool replace_container_info)
+			     string &extra, bool replace_container_info,
+			     falco_common::priority_type min_priority)
 {
 	lua_getglobal(m_ls, m_lua_load_rules.c_str());
 	if(lua_isfunction(m_ls, -1))
@@ -221,7 +222,8 @@ void falco_rules::load_rules(const string &rules_content,
 		lua_pushboolean(m_ls, (all_events ? 1 : 0));
 		lua_pushstring(m_ls, extra.c_str());
 		lua_pushboolean(m_ls, (replace_container_info ? 1 : 0));
-		if(lua_pcall(m_ls, 6, 0, 0) != 0)
+		lua_pushnumber(m_ls, min_priority);
+		if(lua_pcall(m_ls, 7, 0, 0) != 0)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Error loading rules:" + string(lerr);

userspace/engine/rules.h

@@ -24,6 +24,8 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 
 #include "lua_parser.h"
 
+#include "falco_common.h"
+
 class falco_engine;
 
 class falco_rules
@@ -32,7 +34,8 @@ class falco_rules
 	falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls);
 	~falco_rules();
 	void load_rules(const string &rules_content, bool verbose, bool all_events,
-			std::string &extra, bool replace_container_info);
+			std::string &extra, bool replace_container_info,
+			falco_common::priority_type min_priority);
 	void describe_rule(string *rule);
 
 	static void init(lua_State *ls);

userspace/falco/CMakeLists.txt

@@ -19,7 +19,7 @@ target_link_libraries(falco
 
 configure_file(config_falco.h.in config_falco.h)
 
-install(TARGETS falco DESTINATION bin)
+install(TARGETS falco DESTINATION ${FALCO_BIN_DIR})
 install(DIRECTORY lua
-	DESTINATION share/falco
+	DESTINATION ${FALCO_SHARE_DIR}
 	FILES_MATCHING PATTERN *.lua)

userspace/falco/config_falco.h.in

@@ -5,7 +5,7 @@
 #define FALCO_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/"
 #define FALCO_SOURCE_DIR "${PROJECT_SOURCE_DIR}"
 #define FALCO_SOURCE_CONF_FILE "${PROJECT_SOURCE_DIR}/falco.yaml"
-#define FALCO_INSTALL_CONF_FILE "/etc/falco.yaml"
+#define FALCO_INSTALL_CONF_FILE "/etc/falco/falco.yaml"
 #define FALCO_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/userspace/falco/lua/"
 
 #define PROBE_NAME "${PROBE_NAME}"

userspace/falco/configuration.cpp

@@ -22,7 +22,8 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 using namespace std;
 
 falco_configuration::falco_configuration()
-	: m_config(NULL)
+	: m_buffered_outputs(true),
+	  m_config(NULL)
 {
 }
 
@@ -51,20 +52,37 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 
 	init_cmdline_options(cmdline_options);
 
-	m_rules_filenames.push_back(m_config->get_scalar<string>("rules_file", "/etc/falco_rules.yaml"));
+	list<string> rules_files;
+
+	m_config->get_sequence<list<string>>(rules_files, string("rules_file"));
+
+	for(auto &file : rules_files)
+	{
+		// Here, we only include files that exist
+		struct stat buffer;
+		if(stat(file.c_str(), &buffer) == 0)
+		{
+			m_rules_filenames.push_back(file);
+		}
+	}
+
 	m_json_output = m_config->get_scalar<bool>("json_output", false);
 
 	falco_outputs::output_config file_output;
 	file_output.name = "file";
 	if (m_config->get_scalar<bool>("file_output", "enabled", false))
 	{
-		string filename;
+		string filename, keep_alive;
 		filename = m_config->get_scalar<string>("file_output", "filename", "");
 		if (filename == string(""))
 		{
 			throw invalid_argument("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block");
 		}
 		file_output.options["filename"] = filename;
+
+		keep_alive = m_config->get_scalar<string>("file_output", "keep_alive", "");
+		file_output.options["keep_alive"] = keep_alive;
+
 		m_outputs.push_back(file_output);
 	}
 
@@ -86,13 +104,17 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 	program_output.name = "program";
 	if (m_config->get_scalar<bool>("program_output", "enabled", false))
 	{
-		string program;
+		string program, keep_alive;
 		program = m_config->get_scalar<string>("program_output", "program", "");
 		if (program == string(""))
 		{
 			throw sinsp_exception("Error reading config file (" + m_config_file + "): program output enabled but no program in configuration block");
 		}
 		program_output.options["program"] = program;
+
+		keep_alive = m_config->get_scalar<string>("program_output", "keep_alive", "");
+		program_output.options["keep_alive"] = keep_alive;
+
 		m_outputs.push_back(program_output);
 	}
 
@@ -108,6 +130,21 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 	m_notifications_rate = m_config->get_scalar<uint32_t>("outputs", "rate", 1);
 	m_notifications_max_burst = m_config->get_scalar<uint32_t>("outputs", "max_burst", 1000);
 
+	string priority = m_config->get_scalar<string>("priority", "debug");
+	vector<string>::iterator it;
+
+	auto comp = [priority] (string &s) {
+		return (strcasecmp(s.c_str(), priority.c_str()) == 0);
+	};
+
+	if((it = std::find_if(falco_common::priority_names.begin(), falco_common::priority_names.end(), comp)) == falco_common::priority_names.end())
+	{
+		throw invalid_argument("Unknown priority \"" + priority + "\"--must be one of emergency, alert, critical, error, warning, notice, informational, debug");
+	}
+	m_min_priority = (falco_common::priority_type) (it - falco_common::priority_names.begin());
+
+	m_buffered_outputs = m_config->get_scalar<bool>("buffered_outputs", true);
+
 	falco_logger::log_stderr = m_config->get_scalar<bool>("log_stderr", false);
 	falco_logger::log_syslog = m_config->get_scalar<bool>("log_syslog", true);
 }

userspace/falco/configuration.h

@@ -18,6 +18,9 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 
 #pragma once
 
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
 #include <yaml-cpp/yaml.h>
 #include <string>
 #include <vector>
@@ -127,6 +130,27 @@ public:
 		}
 	}
 
+	// called with the last variadic arg (where the sequence is expected to be found)
+	template <typename T>
+	void get_sequence(T& ret, const std::string& name)
+	{
+		YAML::Node child_node = m_root[name];
+		if(child_node.IsDefined())
+		{
+			if(child_node.IsSequence())
+			{
+				for(const YAML::Node& item : child_node)
+				{
+					ret.insert(ret.end(), item.as<typename T::value_type>());
+				}
+			}
+			else if(child_node.IsScalar())
+			{
+				ret.insert(ret.end(), child_node.as<typename T::value_type>());
+			}
+		}
+	}
+
 private:
 	YAML::Node m_root;
 };
@@ -146,6 +170,10 @@ class falco_configuration
 	std::vector<falco_outputs::output_config> m_outputs;
 	uint32_t m_notifications_rate;
 	uint32_t m_notifications_max_burst;
+
+	falco_common::priority_type m_min_priority;
+
+	bool m_buffered_outputs;
  private:
 	void init_cmdline_options(std::list<std::string> &cmdline_options);
 

userspace/falco/falco.cpp

@@ -107,6 +107,11 @@ static void usage()
 	   "                               Can not be specified with -t.\n"
 	   " -t <tag>                      Only run those rules with a tag=<tag>. Can be specified multiple times.\n"
 	   "                               Can not be specified with -T/-D.\n"
+	   " -U,--unbuffered               Turn off output buffering to configured outputs. This causes every\n"
+	   "                               single line emitted by falco to be flushed, which generates higher CPU\n"
+	   "                               usage but is useful when piping those outputs into another process\n"
+	   "                               or into a script.\n"
+	   " -V,--validate <rules_file>    Read the contents of the specified rules file and exit\n"
 	   " -v                            Verbose output.\n"
            " --version                     Print version number.\n"
 	   "\n"
@@ -212,7 +217,7 @@ uint64_t do_inspect(falco_engine *engine,
 		unique_ptr<falco_engine::rule_result> res = engine->process_event(ev);
 		if(res)
 		{
-			outputs->handle_event(res->evt, res->rule, res->priority, res->format);
+			outputs->handle_event(res->evt, res->rule, res->priority_num, res->format);
 		}
 
 		num_evts++;
@@ -240,6 +245,7 @@ int falco_init(int argc, char **argv)
 	string pidfilename = "/var/run/falco.pid";
 	bool describe_all_rules = false;
 	string describe_rule = "";
+	string validate_rules_file = "";
 	string stats_filename = "";
 	bool verbose = false;
 	bool all_events = false;
@@ -256,6 +262,8 @@ int falco_init(int argc, char **argv)
 	int file_limit = 0;
 	unsigned long event_limit = 0L;
 	bool compress = false;
+	bool buffered_outputs = true;
+	bool buffered_cmdline = false;
 
 	// Used for stats
 	uint64_t num_evts;
@@ -272,7 +280,9 @@ int falco_init(int argc, char **argv)
 		{"option", required_argument, 0, 'o'},
 		{"print", required_argument, 0, 'p' },
 		{"pidfile", required_argument, 0, 'P' },
+		{"unbuffered", no_argument, 0, 'U' },
 		{"version", no_argument, 0, 0 },
+		{"validate", required_argument, 0, 0 },
 		{"writefile", required_argument, 0, 'w' },
 
 		{0, 0, 0, 0}
@@ -290,7 +300,7 @@ int falco_init(int argc, char **argv)
 		// Parse the args
 		//
 		while((op = getopt_long(argc, argv,
-                                        "hc:AdD:e:k:K:Ll:m:M:o:P:p:r:s:T:t:vw:",
+                                        "hc:AdD:e:k:K:Ll:m:M:o:P:p:r:s:T:t:UvV:w:",
                                         long_options, &long_index)) != -1)
 		{
 			switch(op)
@@ -378,9 +388,16 @@ int falco_init(int argc, char **argv)
 			case 't':
 				enabled_rule_tags.insert(optarg);
 				break;
+			case 'U':
+				buffered_outputs = false;
+				buffered_cmdline = true;
+				break;
 			case 'v':
 				verbose = true;
 				break;
+			case 'V':
+				validate_rules_file = optarg;
+				break;
 			case 'w':
 				outfile = optarg;
 				break;
@@ -443,6 +460,14 @@ int falco_init(int argc, char **argv)
 			}
 		}
 
+		if(validate_rules_file != "")
+		{
+			falco_logger::log(LOG_INFO, "Validating rules file: " + validate_rules_file + "...\n");
+			engine->load_rules_file(validate_rules_file, verbose, all_events);
+			falco_logger::log(LOG_INFO, "Ok\n");
+			goto exit;
+		}
+
 		falco_configuration config;
 		if (conf_filename.size())
 		{
@@ -461,6 +486,18 @@ int falco_init(int argc, char **argv)
 			config.m_rules_filenames = rules_filenames;
 		}
 
+		engine->set_min_priority(config.m_min_priority);
+
+		if(buffered_cmdline)
+		{
+			config.m_buffered_outputs = buffered_outputs;
+		}
+
+		if(config.m_rules_filenames.size() == 0)
+		{
+			throw std::invalid_argument("You must specify at least one rules file via -r or a rules_file entry in falco.yaml");
+		}
+
 		for (auto filename : config.m_rules_filenames)
 		{
 			engine->load_rules_file(filename, verbose, all_events);
@@ -501,7 +538,9 @@ int falco_init(int argc, char **argv)
 			engine->enable_rule_by_tag(enabled_rule_tags, true);
 		}
 
-		outputs->init(config.m_json_output, config.m_notifications_rate, config.m_notifications_max_burst);
+		outputs->init(config.m_json_output,
+			      config.m_notifications_rate, config.m_notifications_max_burst,
+			      config.m_buffered_outputs);
 
 		if(!all_events)
 		{

userspace/falco/falco_outputs.cpp

@@ -27,7 +27,8 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 using namespace std;
 
 falco_outputs::falco_outputs()
-	: m_initialized(false)
+	: m_initialized(false),
+	  m_buffered(true)
 {
 
 }
@@ -51,7 +52,7 @@ falco_outputs::~falco_outputs()
 	}
 }
 
-void falco_outputs::init(bool json_output, uint32_t rate, uint32_t max_burst)
+void falco_outputs::init(bool json_output, uint32_t rate, uint32_t max_burst, bool buffered)
 {
 	// The engine must have been given an inspector by now.
 	if(! m_inspector)
@@ -70,12 +71,14 @@ void falco_outputs::init(bool json_output, uint32_t rate, uint32_t max_burst)
 
 	m_notifications_tb.init(rate, max_burst);
 
+	m_buffered = buffered;
+
 	m_initialized = true;
 }
 
 void falco_outputs::add_output(output_config oc)
 {
-	uint8_t nargs = 1;
+	uint8_t nargs = 2;
 	lua_getglobal(m_ls, m_lua_add_output.c_str());
 
 	if(!lua_isfunction(m_ls, -1))
@@ -83,11 +86,12 @@ void falco_outputs::add_output(output_config oc)
 		throw falco_exception("No function " + m_lua_add_output + " found. ");
 	}
 	lua_pushstring(m_ls, oc.name.c_str());
+	lua_pushnumber(m_ls, (m_buffered ? 1 : 0));
 
 	// If we have options, build up a lua table containing them
 	if (oc.options.size())
 	{
-		nargs = 2;
+		nargs = 3;
 		lua_createtable(m_ls, 0, oc.options.size());
 
 		for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it)
@@ -105,7 +109,7 @@ void falco_outputs::add_output(output_config oc)
 
 }
 
-void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority, string &format)
+void falco_outputs::handle_event(sinsp_evt *ev, string &rule, falco_common::priority_type priority, string &format)
 {
 	if(!m_notifications_tb.claim())
 	{
@@ -119,10 +123,11 @@ void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority,
 	{
 		lua_pushlightuserdata(m_ls, ev);
 		lua_pushstring(m_ls, rule.c_str());
-		lua_pushstring(m_ls, priority.c_str());
+		lua_pushstring(m_ls, falco_common::priority_names[priority].c_str());
+		lua_pushnumber(m_ls, priority);
 		lua_pushstring(m_ls, format.c_str());
 
-		if(lua_pcall(m_ls, 4, 0, 0) != 0)
+		if(lua_pcall(m_ls, 5, 0, 0) != 0)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Error invoking function output: " + string(lerr);

userspace/falco/falco_outputs.h

@@ -41,7 +41,7 @@ public:
 		std::map<std::string, std::string> options;
 	};
 
-	void init(bool json_output, uint32_t rate, uint32_t max_burst);
+	void init(bool json_output, uint32_t rate, uint32_t max_burst, bool buffered);
 
 	void add_output(output_config oc);
 
@@ -49,7 +49,7 @@ public:
 	// ev is an event that has matched some rule. Pass the event
 	// to all configured outputs.
 	//
-	void handle_event(sinsp_evt *ev, std::string &rule, std::string &priority, std::string &format);
+	void handle_event(sinsp_evt *ev, std::string &rule, falco_common::priority_type priority, std::string &format);
 
 private:
 	bool m_initialized;
@@ -57,6 +57,8 @@ private:
 	// Rate limits notifications
 	token_bucket m_notifications_tb;
 
+	bool m_buffered;
+
 	std::string m_lua_add_output = "add_output";
 	std::string m_lua_output_event = "output_event";
 	std::string m_lua_output_cleanup = "output_cleanup";

userspace/falco/lua/output.lua

@@ -18,22 +18,25 @@
 
 local mod = {}
 
-levels = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"}
-
-mod.levels = levels
-
 local outputs = {}
 
-function mod.stdout(level, msg)
+function mod.stdout(priority, priority_num, buffered, msg)
+   if buffered == 0 then
+      io.stdout:setvbuf 'no'
+   end
    print (msg)
 end
 
+function mod.stdout_cleanup()
+   io.stdout:flush()
+end
+
 function mod.file_validate(options)
    if (not type(options.filename) == 'string') then
       error("File output needs to be configured with a valid filename")
    end
 
-   file, err = io.open(options.filename, "a+")
+   local file, err = io.open(options.filename, "a+")
    if file == nil then
       error("Error with file output: "..err)
    end
@@ -41,60 +44,100 @@ function mod.file_validate(options)
 
 end
 
-function mod.file(level, msg, options)
-   file = io.open(options.filename, "a+")
+function mod.file(priority, priority_num, buffered, msg, options)
+   if options.keep_alive == "true" then
+      if file == nil then
+	 file = io.open(options.filename, "a+")
+	 if buffered == 0 then
+	    file:setvbuf 'no'
+	 end
+      end
+   else
+      file = io.open(options.filename, "a+")
+   end
+
    file:write(msg, "\n")
-   file:close()
+
+   if options.keep_alive == nil or
+      options.keep_alive ~= "true" then
+	 file:close()
+	 file = nil
+   end
 end
 
-function mod.syslog(level, msg, options)
-   falco.syslog(level, msg)
+function mod.file_cleanup()
+   if file ~= nil then
+      file:flush()
+      file:close()
+      file = nil
+   end
 end
 
-function mod.program(level, msg, options)
+function mod.syslog(priority, priority_num, buffered, msg, options)
+   falco.syslog(priority_num, msg)
+end
+
+function mod.syslog_cleanup()
+end
+
+function mod.program(priority, priority_num, buffered, msg, options)
    -- XXX Ideally we'd check that the program ran
    -- successfully. However, the luajit we're using returns true even
    -- when the shell can't run the program.
 
-   file = io.popen(options.program, "w")
+   -- Note: options are all strings
+   if options.keep_alive == "true" then
+      if file == nil then
+	 file = io.popen(options.program, "w")
+	 if buffered == 0 then
+	    file:setvbuf 'no'
+	 end
+      end
+   else
+      file = io.popen(options.program, "w")
+   end
 
    file:write(msg, "\n")
-   file:close()
-end
 
-local function level_of(s)
-   s = string.lower(s)
-   for i,v in ipairs(levels) do
-      if (string.find(string.lower(v), "^"..s)) then
-	 return i - 1 -- (syslog levels start at 0, lua indices start at 1)
-      end
+   if options.keep_alive == nil or
+      options.keep_alive ~= "true" then
+	 file:close()
+	 file = nil
    end
-   error("Invalid severity level: "..s)
 end
 
-function output_event(event, rule, priority, format)
-   local level = level_of(priority)
+function mod.program_cleanup()
+   if file ~= nil then
+      file:flush()
+      file:close()
+      file = nil
+   end
+end
 
+function output_event(event, rule, priority, priority_num, format)
    -- If format starts with a *, remove it, as we're adding our own
    -- prefix here.
    if format:sub(1,1) == "*" then
       format = format:sub(2)
    end
 
-   format = "*%evt.time: "..levels[level+1].." "..format
+   format = "*%evt.time: "..priority.." "..format
 
-   msg = formats.format_event(event, rule, levels[level+1], format)
+   msg = formats.format_event(event, rule, priority, format)
 
    for index,o in ipairs(outputs) do
-      o.output(level, msg, o.config)
+      o.output(priority, priority_num, o.buffered, msg, o.config)
    end
 end
 
 function output_cleanup()
    formats.free_formatters()
+   for index,o in ipairs(outputs) do
+      o.cleanup()
+   end
 end
 
-function add_output(output_name, config)
+function add_output(output_name, buffered, config)
    if not (type(mod[output_name]) == 'function') then
       error("rule_loader.add_output(): invalid output_name: "..output_name)
    end
@@ -105,7 +148,9 @@ function add_output(output_name, config)
      mod[output_name.."_validate"](config)
    end
 
-   table.insert(outputs, {output = mod[output_name], config=config})
+   table.insert(outputs, {output = mod[output_name],
+			  cleanup = mod[output_name.."_cleanup"],
+			  buffered=buffered, config=config})
 end
 
 return mod