Refactor shell rules (#301)

* Refactor shell rules to avoid FPs.

Refactoring the shell related rules to avoid FPs. Instead of considering
all shells suspicious and trying to carve out exceptions for the
legitimate uses of shells, only consider shells spawned below certain
processes suspicious.

The set of processes is a collection of commonly used web servers,
databases, nosql document stores, mail programs, message queues, process
monitors, application servers, etc.

Also, runsv is also considered a top level process that denotes a
service. This allows a way for more flexible servers like ad-hoc nodejs
express apps, etc to denote themselves as a full server process.

* Update event generator to reflect new shell rules

spawn_shell is now a silent action. its replacement is
spawn_shell_under_httpd, which respawns itself as httpd and then runs a
shell.

db_program_spawn_binaries now runs ls instead of a shell so it only
matches db_program_spawn_process.

* Comment out old shell related rules

* Modify nodejs example to work w/ new shell rules

Start the express server using runit's runsv, which allows falco to
consider any shells run by it as suspicious.

* Use the updated argument for mkdir

In https://github.com/draios/sysdig/pull/757 the path argument for mkdir
moved to the second argument. This only became visible in the unit tests
once the trace files were updated to reflect the other shell rule
changes--the trace files had the old format.

* Update unit tests for shell rules changes

Shell in container doesn't exist any longer and its functionality has
been subsumed by run shell untrusted.

* Allow git binaries to run shells

In some cases, these are run below a service runsv so we still need
exceptions for them.

* Let consul agent spawn curl for health checks

* Don't protect tomcat

There's enough evidence of people spawning general commands that we
can't protect it.

* Reorder exceptions, add rabbitmq exception

Move the nginx exception to the main rule instead of the
protected_shell_spawner macro. Also add erl_child_setup (related to
rabbitmq) as an allowed shell spawner.

* Add additional spawn binaries

All off these are either below nginx, httpd, or runsv but should still
be allowed to spawn shells.

* Exclude shells when ancestor is a pkg mgmt binary

Skip shells when any process ancestor (parent, gparent, etc) is a
package management binary. This includes the program needrestart. This
is a deep search but should prevent a lot of other more detailed
exceptions trying to find the specific scripts run as a part of
installations.

* Skip shells related to serf

Serf is a service discovery tool and can in some cases be spawned by
apache/nginx. Also allow shells that are just checking the status of
pids via kill -0.

* Add several exclusions back

Add several exclusions back from the shell in container rule. These are
all allowed shell spawns that happen to be below
nginx/fluentd/apache/etc.

* Remove commented-out rules

This saves space as well as cleanup. I haven't yet removed the
macros/lists used by these rules and not used anywhere else. I'll do
that cleanup in a separate step.

* Also exclude based on command lines

Add back the exclusions based on command lines, using the existing set
of command lines.

* Add addl exclusions for shells

Of note is runsv, which means it can directly run shells (the ./run and
./finish scripts), but the things it runs can not.

* Don't trigger on shells spawning shells

We'll detect the first shell and not any other shells it spawns.

* Allow "runc:" parents to count as a cont entrypnt

In some cases, the initial process for a container can have a parent
"runc:[0:PARENT]", so also allow those cases to count as a container
entrypoint.

* Use container_entrypoint macro

Use the container_entrypoint macro to denote entering a container and
also allow exe to be one of the processes that's the parent of an
entrypoint.

docker/event-generator/event_generator.cpp

@@ -50,6 +50,8 @@ void usage(char *program)
 	printf("                                                     then read a sensitive file\n");
 	printf("          write_rpm_database                         Write to files below /var/lib/rpm\n");
 	printf("          spawn_shell                                Run a shell (bash)\n");
+	printf("                                                     Used by spawn_shell_under_httpd below\n");
+	printf("          spawn_shell_under_httpd                    Run a shell (bash) under a httpd process\n");
 	printf("          db_program_spawn_process                   As a database program, try to spawn\n");
 	printf("                                                     another program\n");
 	printf("          modify_binary_dirs                         Modify a file below /bin\n");
@@ -64,7 +66,7 @@ void usage(char *program)
 	printf("          non_sudo_setuid                            Setuid as a non-root user\n");
 	printf("          create_files_below_dev                     Create files below /dev\n");
 	printf("          exec_ls                                    execve() the program ls\n");
-	printf("                                                     (used by user_mgmt_binaries below)\n");
+	printf("                                                     (used by user_mgmt_binaries, db_program_spawn_process)\n");
 	printf("          user_mgmt_binaries                         Become the program \"vipw\", which triggers\n");
 	printf("                                                     rules related to user management programs\n");
 	printf("          exfiltration                               Read /etc/shadow and send it via udp to a\n");
@@ -230,9 +232,14 @@ void spawn_shell() {
 	}
 }
 
+void spawn_shell_under_httpd() {
+	printf("Becoming the program \"httpd\" and then spawning a shell\n");
+	respawn("./httpd", "spawn_shell", "0");
+}
+
 void db_program_spawn_process() {
-	printf("Becoming the program \"mysql\" and then spawning a shell\n");
-	respawn("./mysqld", "spawn_shell", "0");
+	printf("Becoming the program \"mysql\" and then running ls\n");
+	respawn("./mysqld", "exec_ls", "0");
 }
 
 void modify_binary_dirs() {
@@ -360,6 +367,7 @@ map<string, action_t> defined_actions = {{"write_binary_dir", write_binary_dir},
 					 {"read_sensitive_file_after_startup", read_sensitive_file_after_startup},
 					 {"write_rpm_database", write_rpm_database},
 					 {"spawn_shell", spawn_shell},
+					 {"spawn_shell_under_httpd", spawn_shell_under_httpd},
 					 {"db_program_spawn_process", db_program_spawn_process},
 					 {"modify_binary_dirs", modify_binary_dirs},
 					 {"mkdir_binary_dirs", mkdir_binary_dirs},
@@ -375,7 +383,7 @@ map<string, action_t> defined_actions = {{"write_binary_dir", write_binary_dir},
 
 // Some actions don't directly result in suspicious behavior. These
 // actions are excluded from the ones run with -a all.
-set<string> exclude_from_all_actions = {"exec_ls", "network_activity"};
+set<string> exclude_from_all_actions = {"spawn_shell", "exec_ls", "network_activity"};
 
 void create_symlinks(const char *program)
 {

examples/nodejs-bad-rest-api/demo.yml

@@ -1,9 +1,7 @@
-# Owned by software vendor, serving install-software.sh.
 express_server:
   container_name: express_server
   image: node:latest
-  working_dir: /usr/src/app
-  command: bash -c "npm install && node server.js"
+  command: bash -c "apt-get -y update && apt-get -y install runit && npm install && runsv /usr/src/app"
   ports:
     - "8181:8181"
   volumes:

examples/nodejs-bad-rest-api/run

@@ -0,0 +1,2 @@
+#!/bin/sh
+node server.js

rules/falco_rules.yaml

@@ -41,10 +41,10 @@
 
 - macro: bin_dir_mkdir
   condition: >
-    evt.arg[0] startswith /bin/ or
-    evt.arg[0] startswith /sbin/ or
-    evt.arg[0] startswith /usr/bin/ or
-    evt.arg[0] startswith /usr/sbin/
+    (evt.arg[1] startswith /bin/ or
+     evt.arg[1] startswith /sbin/ or
+     evt.arg[1] startswith /usr/bin/ or
+     evt.arg[1] startswith /usr/sbin/)
 
 - macro: bin_dir_rename
   condition: >
@@ -156,10 +156,10 @@
   items: [chef-client]
 
 - list: http_server_binaries
-  items: [nginx, httpd, httpd-foregroun, lighttpd]
+  items: [nginx, httpd, httpd-foregroun, lighttpd, apache, apache2]
 
 - list: db_server_binaries
-  items: [mysqld]
+  items: [mysqld, postgres, sqlplus]
 
 - list: mysql_mgmt_binaries
   items: [mysql_install_d, mysql_ssl_rsa_s]
@@ -170,6 +170,9 @@
 - list: db_mgmt_binaries
   items: [mysql_mgmt_binaries, postgres_mgmt_binaries]
 
+- list: nosql_server_binaries
+  items: [couchdb, memcached, redis-server, rabbitmq-server, mongod]
+
 - list: gitlab_binaries
   items: [gitlab-shell, gitlab-mon, gitlab-runner-b, git]
 
@@ -199,6 +202,9 @@
 - macro: package_mgmt_procs
   condition: proc.name in (package_mgmt_binaries)
 
+- macro: run_by_package_mgmt_binaries
+  condition: proc.aname in (package_mgmt_binaries, needrestart)
+
 - list: ssl_mgmt_binaries
   items: [ca-certificates]
 
@@ -562,9 +568,6 @@
 - macro: parent_java_running_confluence
   condition: (proc.pname=java and proc.pcmdline contains "-classpath /opt/atlassian/confluence")
 
-- macro: parent_java_running_tomcat
-  condition: (proc.pname=java and proc.pcmdline contains "-classpath /usr/local/tomcat")
-
 - macro: parent_java_running_install4j
   condition: (proc.pname=java and proc.pcmdline contains "-classpath i4jruntime.jar")
 
@@ -989,66 +992,104 @@
     mysql_upgrade, opkg-cl, vmtoolsd, confd
     ]
 
-- rule: Run shell untrusted
-  desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries.
+# The binaries in this list and their descendents are *not* allowed
+# spawn shells. This includes the binaries spawning shells directly as
+# well as indirectly. For example, apache -> php/perl for
+# mod_{php,perl} -> some shell is also not allowed, because the shell
+# has apache as an ancestor.
+
+- list: protected_shell_spawning_binaries
+  items: [
+    http_server_binaries, db_server_binaries, nosql_server_binaries, mail_binaries,
+    fluentd, flanneld, splunkd, consul, runsv
+    ]
+
+- macro: parent_java_running_zookeeper
+  condition: (proc.pname=java and proc.pcmdline contains org.apache.zookeeper.server)
+
+- macro: parent_java_running_kafka
+  condition: (proc.pname=java and proc.pcmdline contains kafka.Kafka)
+
+- macro: parent_java_running_elasticsearch
+  condition: (proc.pname=java and proc.pcmdline contains org.elasticsearch.bootstrap.Elasticsearch)
+
+- macro: parent_java_running_activemq
+  condition: (proc.pname=java and proc.pcmdline contains activemq.jar)
+
+- macro: parent_java_running_cassandra
+  condition: (proc.pname=java and proc.pcmdline contains org.apache.cassandra.service.CassandraDaemon)
+
+- macro: parent_java_running_jboss_wildfly
+  condition: (proc.pname=java and proc.pcmdline contains org.jboss)
+
+- macro: parent_java_running_glassfish
+  condition: (proc.pname=java and proc.pcmdline contains com.sun.enterprise.glassfish)
+
+- macro: parent_java_running_hadoop
+  condition: (proc.pname=java and proc.pcmdline contains org.apache.hadoop)
+
+- macro: parent_java_running_datastax
+  condition: (proc.pname=java and proc.pcmdline contains com.datastax)
+
+- macro: parent_java_running_sumologic
+  condition: (proc.pname=java and proc.pcmdline contains com.sumologic)
+
+- macro: nginx_starting_nginx
+  condition: (proc.pname=nginx and proc.cmdline contains "/usr/sbin/nginx -c /etc/nginx/nginx.conf")
+
+- macro: consul_running_curl
+  condition: (proc.pname=consul and proc.cmdline startswith "sh -c curl")
+
+- macro: serf_script
+  condition: (proc.cmdline startswith "sh -c serf")
+
+- macro: check_process_status
+  condition: (proc.cmdline startswith "sh -c kill -0 ")
+
+- macro: protected_shell_spawner
   condition: >
-    spawned_process and not container
+    (proc.aname in (protected_shell_spawning_binaries)
+    or parent_java_running_zookeeper
+    or parent_java_running_kafka
+    or parent_java_running_elasticsearch
+    or parent_java_running_activemq
+    or parent_java_running_cassandra
+    or parent_java_running_jboss_wildfly
+    or parent_java_running_glassfish
+    or parent_java_running_hadoop
+    or parent_java_running_datastax)
+
+# Note that runsv is both in protected_shell_spawner and the
+# exclusions by pname. This means that runsv can itself spawn shells
+# (the ./run and ./finish scripts), but the processes runsv can not
+# spawn shells.
+- rule: Run shell untrusted
+  desc: an attempt to spawn a shell below a non-shell application. Specific applications are monitored.
+  condition: >
+    spawned_process
     and shell_procs
     and proc.pname exists
-    and not proc.pname in (cron_binaries, shell_binaries, make_binaries, known_shell_spawn_binaries, docker_binaries,
-                           k8s_binaries, package_mgmt_binaries, aide_wrapper_binaries, nids_binaries,
-                           monitoring_binaries, gitlab_binaries, mesos_slave_binaries,
-                           keepalived_binaries,
-                           needrestart_binaries, phusion_passenger_binaries, chef_binaries, nomachine_binaries,
-                           x2go_binaries, db_mgmt_binaries, plesk_binaries)
-    and not parent_ansible_running_python
-    and not parent_bro_running_python
-    and not parent_python_running_denyhosts
-    and not parent_python_running_sdchecks
-    and not parent_linux_image_upgrade_script
-    and not parent_java_running_jenkins
+    and protected_shell_spawner
+    and not proc.pname in (shell_binaries, gitlab_binaries, cron_binaries,
+                           erl_child_setup, exechealthz,
+                           PM2, PassengerWatchd, c_rehash, svlogd, logrotate, hhvm, serf,
+                           lb-controller, nvidia-installe, runsv, statsite)
     and not proc.cmdline in (known_shell_spawn_cmdlines)
-    and not jenkins_scripts
-    and not parent_java_running_echo
-    and not parent_scripting_running_builds
-    and not makefile_perl
-    and not parent_Xvfb_running_xkbcomp
-    and not parent_nginx_running_serf
-    and not parent_node_running_npm
-    and not parent_npm_running_node
-    and not parent_java_running_sbt
-    and not parent_beam_running_python
-    and not parent_strongswan_running_starter
-    and not run_by_chef
-    and not run_by_puppet
-    and not run_by_adclient
-    and not run_by_centrify
-    and not parent_dovecot_running_auth
+    and not proc.aname in (unicorn_launche)
+    and not consul_running_curl
+    and not nginx_starting_nginx
+    and not run_by_package_mgmt_binaries
+    and not serf_script
+    and not check_process_status
     and not run_by_foreman
-    and not run_by_openshift
-    and not parent_java_running_tomcat
-    and not parent_java_running_install4j
-    and not parent_java_running_endeca
-    and not parent_running_datastax
-    and not parent_java_running_appdynamics
-    and not parent_cpanm_running_perl
-    and not parent_ruby_running_discourse
-    and not parent_ruby_running_pups
-    and not assemble_running_php
-    and not node_running_bitnami
-    and not node_running_threatstack
-    and not parent_python_running_localstack
-    and not parent_python_running_zookeeper
-    and not parent_python_running_airflow
-    and not perl_running_plesk
-    and not plesk_autoinstaller
-    and not parent_perl_running_openresty
+    and not python_mesos_marathon_scripting
+    and not user_shell_container_exclusions
   output: >
     Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname
     cmdline=%proc.cmdline pcmdline=%proc.pcmdline gparent=%proc.aname[2] ggparent=%proc.aname[3]
     gggparent=%proc.aname[4] ggggparent=%proc.aname[5])
   priority: DEBUG
-  tags: [host, shell]
+  tags: [shell]
 
 - macro: trusted_containers
   condition: (container.image startswith sysdig/agent or
@@ -1122,7 +1163,7 @@
 # when we lose events and lose track of state.
 
 - macro: container_entrypoint
-  condition: (not proc.pname exists or proc.pname in (runc:[0:PARENT], runc:[1:CHILD], docker-runc))
+  condition: (not proc.pname exists or proc.pname in (runc:[0:PARENT], runc:[1:CHILD], docker-runc, exe))
 
 - rule: Launch Sensitive Mount Container
   desc: >
@@ -1171,11 +1212,11 @@
   tags: [users]
 
 - rule: Terminal shell in container
-  desc: A shell was spawned by a program in a container with an attached terminal.
+  desc: A shell was used as the entrypoint/exec point into a container with an attached terminal.
   condition: >
     spawned_process and container
     and shell_procs and proc.tty != 0
-    and not proc.cmdline in (known_shell_spawn_cmdlines)
+    and container_entrypoint
   output: >
     A shell was spawned in a container with an attached terminal (user=%user.name %container.info
     shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty)
@@ -1262,84 +1303,6 @@
     (proc.pname=node and (proc.pcmdline contains /var/www/edi/process.js or
                           proc.pcmdline contains "sh -c /var/www/edi/bin/sftp.sh"))
 
-- rule: Run shell in container
-  desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded.
-  condition: >
-    spawned_process and container
-    and shell_procs
-    and not container_entrypoint
-    and not proc.pname in (shell_binaries, make_binaries, docker_binaries, k8s_binaries, package_mgmt_binaries,
-                           lxd_binaries, mesos_slave_binaries, aide_wrapper_binaries, nids_binaries,
-                           cron_binaries,
-                           user_known_container_shell_spawn_binaries,
-                           needrestart_binaries,
-                           phusion_passenger_binaries,
-                           chef_binaries,
-                           nomachine_binaries,
-                           x2go_binaries,
-                           db_mgmt_binaries,
-                           plesk_binaries,
-                           monitoring_binaries, gitlab_binaries, initdb, awk, falco, cron,
-                           erl_child_setup, erlexec, ceph, PM2, pycompile, py3compile, hhvm, npm, serf,
-                           runsv, supervisord, varnishd, crond, logrotate, timeout, tini,
-                           xrdb, xfce4-session, weave, logdna-agent, bundle, configure, luajit, nginx,
-                           beam.smp, paster, postfix-local, hawkular-metric, fluentd, x2gormforward,
-                           "[celeryd:", flock, nsrun, consul, migrate-databas, airflow, bootstrap-qmf-l,
-                           build-qmf-artif, colormake.pl, doxygen, Cypress, lb-controller, vmtoolsd,
-                           haproxy_reload., curator, consul-template, xargs, scl, find, awstats_updatea,
-                           sa-update, mysql_upgrade, opkg-cl, peer-finder, confd, aws)
-    and not trusted_containers
-    and not shell_spawning_containers
-    and not parent_java_running_echo
-    and not parent_scripting_running_builds
-    and not makefile_perl
-    and not parent_Xvfb_running_xkbcomp
-    and not mysql_image_running_healthcheck
-    and not parent_nginx_running_serf
-    and not proc.cmdline in (known_container_shell_spawn_cmdlines)
-    and not parent_node_running_npm
-    and not parent_npm_running_node
-    and not user_shell_container_exclusions
-    and not node_running_edi_dynamodb
-    and not run_by_h2o
-    and not run_by_passenger_agent
-    and not parent_java_running_jenkins
-    and not parent_java_running_maven
-    and not parent_java_running_appdynamics
-    and not parent_java_running_sbt
-    and not python_running_es_curator
-    and not parent_beam_running_python
-    and not jenkins_scripts
-    and not bundle_running_ruby
-    and not parent_dovecot_running_auth
-    and not parent_strongswan_running_starter
-    and not parent_phusion_passenger_my_init
-    and not parent_java_running_confluence
-    and not parent_java_running_tomcat
-    and not parent_java_running_install4j
-    and not parent_running_datastax
-    and not ics_running_java
-    and not parent_ruby_running_discourse
-    and not parent_ruby_running_pups
-    and not assemble_running_php
-    and not node_running_bitnami
-    and not node_running_threatstack
-    and not parent_python_running_localstack
-    and not parent_python_running_zookeeper
-    and not parent_python_running_airflow
-    and not parent_docker_start_script
-    and not parent_java_running_endeca
-    and not python_mesos_healthcheck
-    and not python_mesos_marathon_scripting
-    and not perl_running_plesk
-    and not parent_rancher_running_healthcheck
-    and not parent_perl_running_openresty
-  output: >
-    Shell spawned in a container other than entrypoint (user=%user.name %container.info image=%container.image
-    shell=%proc.name pcmdline=%proc.pcmdline cmdline=%proc.cmdline parent=%proc.pname gparent=%proc.aname[2] ggparent=%proc.aname[3])
-  priority: DEBUG
-  tags: [container, shell]
-
 - macro: login_doing_dns_lookup
   condition: (proc.name=login and fd.l4proto=udp and fd.sport=53)
 

test/falco_tests.yaml

@@ -319,7 +319,7 @@ trace_files: !mux
     detect_counts:
       - "Write below binary dir": 1
       - "Read sensitive file untrusted": 3
-      - "Run shell in container": 1
+      - "Run shell untrusted": 1
       - "Write below rpm database": 1
       - "Write below etc": 1
       - "System procs network activity": 1

test/falco_traces.yaml.in

@@ -43,11 +43,11 @@ traces: !mux
   falco-event-generator:
     trace_file: traces-positive/falco-event-generator.scap
     detect: True
-    detect_level: [ERROR, WARNING, INFO, NOTICE]
+    detect_level: [ERROR, WARNING, INFO, NOTICE, DEBUG]
     detect_counts:
       - "Write below binary dir": 1
       - "Read sensitive file untrusted": 3
-      - "Run shell in container": 1
+      - "Run shell untrusted": 1
       - "Write below rpm database": 1
       - "Write below etc": 1
       - "System procs network activity": 1
@@ -146,13 +146,6 @@ traces: !mux
     detect_counts:
       - "Run shell untrusted": 1
 
-  shell-in-container:
-    trace_file: traces-positive/shell-in-container.scap
-    detect: True
-    detect_level: DEBUG
-    detect_counts:
-      - "Run shell in container": 1
-
   system-binaries-network-activity:
     trace_file: traces-positive/system-binaries-network-activity.scap
     detect: True