Merge branch 'dev' into agent-master

the commit.

CHANGELOG.md

@@ -2,48 +2,6 @@
 
 This file documents all notable changes to Falco. The release numbering uses [semantic versioning](http://semver.org).
 
-## v0.10.0
-
-Released 2018-04-24
-
-## Major Changes
-
-* **Rules Directory Support**: Falco will read rules files from `/etc/falco/rules.d` in addition to `/etc/falco/falco_rules.yaml` and `/etc/falco/falco_rules.local.yaml`. Also, when the argument to `-r`/falco.yaml `rules_file` is a directory, falco will read rules files from that directory. [[#348](https://github.com/draios/falco/pull/348)] [[#187](https://github.com/draios/falco/issues/187)]
-* Properly support all syscalls (e.g. those without parameter extraction by the kernel module) in falco conditions, so they can be included in `evt.type=<name>` conditions. [[#352](https://github.com/draios/falco/pull/352)]
-* When packaged as a container, start building kernel module with gcc 5.0 instead of gcc 4.9. [[#331](https://github.com/draios/falco/pull/331)]
-* New example puppet module for falco. [[#341](https://github.com/draios/falco/pull/341)] [[#115](https://github.com/draios/falco/issues/115)]
-* When signaled with `USR1`, falco will close/reopen log files. Include a [logrotate](https://github.com/logrotate/logrotate) example that shows how to use this feature for log rotation. [[#347](https://github.com/draios/falco/pull/347)] [[#266](https://github.com/draios/falco/issues/266)]
-* To improve resource usage, further restrict the set of system calls available to falco [[#351](https://github.com/draios/falco/pull/351)] [[draios/sysdig#1105](https://github.com/draios/sysdig/pull/1105)]
-
-## Minor Changes
-
-* Add gdb to the development Docker image (sysdig/falco:dev) to aid in debugging. [[#323](https://github.com/draios/falco/pull/323)]
-* You can now specify -V multiple times on the command line to validate multiple rules files at once. [[#329](https://github.com/draios/falco/pull/329)]
-* When run with `-v`, falco will print *dangling* macros/lists that are not used by any rules. [[#329](https://github.com/draios/falco/pull/329)]
-* Add an example demonstrating cryptomining attack that exploits an open docker daemon using host mounts. [[#336](https://github.com/draios/falco/pull/336)]
-* New falco.yaml option `json_include_output_property` controls whether the formatted string "output" is included in the json object when json output is enabled. [[#342](https://github.com/draios/falco/pull/342)]
-* Centralize testing event types for consideration by falco into a single function [[draios/sysdig#1105](https://github.com/draios/sysdig/pull/1105)) [[#356](https://github.com/draios/falco/pull/356)]
-* If a rule has an attribute `warn_evttypes`, falco will not complain about `evt.type` restrictions on that rule [[#355](https://github.com/draios/falco/pull/355)]
-* When run with `-i`, print all ignored events/syscalls and exit. [[#359](https://github.com/draios/falco/pull/359)]
-
-## Bug Fixes
-
-* Minor bug fixes to k8s daemonset configuration. [[#325](https://github.com/draios/falco/pull/325)] [[#296](https://github.com/draios/falco/pull/296)] [[#295](https://github.com/draios/falco/pull/295)]
-* Ensure `--validate` can be used interchangeably with `-V`. [[#334](https://github.com/draios/falco/pull/334)] [[#322](https://github.com/draios/falco/issues/322)]
-* Rule conditions like `fd.net` can now be used with the `in` operator e.g. `evt.type=connect and fd.net in ("127.0.0.1/24")`. [[draios/sysdig#1091](https://github.com/draios/sysdig/pull/1091)] [[#343](https://github.com/draios/falco/pull/343)]
-* Ensure that `keep_alive` can be used both with file and program output at the same time. [[#335](https://github.com/draios/falco/pull/335)]
-* Make it possible to append to a skipped macro/rule without falco complaining [[#346](https://github.com/draios/falco/pull/346)] [[#305](https://github.com/draios/falco/issues/305)]
-* Ensure rule order is preserved even when rules do not contain any `evt.type` restriction. [[#354](https://github.com/draios/falco/issues/354)] [[#355](https://github.com/draios/falco/pull/355)]
-
-## Rule Changes
-
-* Make it easier to extend the `Change thread namespace` rule via a `user_known_change_thread_namespace_binaries` list. [[#324](https://github.com/draios/falco/pull/324)]
-* Various FP fixes from users. [[#321](https://github.com/draios/falco/pull/321)] [[#326](https://github.com/draios/falco/pull/326)] [[#344](https://github.com/draios/falco/pull/344)] [[#350](https://github.com/draios/falco/pull/350)]
-* New rule `Disallowed SSH Connection` detects attempts ssh connection attempts to hosts outside of an expected set. In order to be effective, you need to override the macro `allowed_ssh_hosts` in a user rules file. [[#321](https://github.com/draios/falco/pull/321)]
-* New rule `Unexpected K8s NodePort Connection` detects attempts to contact the K8s NodePort range from a program running inside a container. In order to be effective, you need to override the macro `nodeport_containers` in a user rules file. [[#321](https://github.com/draios/falco/pull/321)]
-* Improve `Modify binary dirs` rule to work with new syscalls [[#353](https://github.com/draios/falco/pull/353)]
-* New rule `Unexpected UDP Traffic` checks for udp traffic not on a list of expected ports. Somewhat FP-prone, so it must be explicitly enabled by overriding the macro `do_unexpected_udp_check` in a user rules file. [[#320](https://github.com/draios/falco/pull/320)] [[#357](https://github.com/draios/falco/pull/357)]
-
 ## v0.9.0
 
 Released 2018-01-18

README.md

@@ -2,7 +2,7 @@
 
 #### Latest release
 
-**v0.10.0**
+**v0.9.0**
 Read the [change log](https://github.com/draios/falco/blob/dev/CHANGELOG.md)
 
 Dev Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=dev)](https://travis-ci.org/draios/falco)<br />

rules/falco_rules.yaml

@@ -15,9 +15,6 @@
 - macro: never_true
   condition: (evt.num=0)
 
-- macro: always_true
-  condition: (evt.num=>0)
-
 # In some cases, such as dropped system call events, information about
 # the process name may be missing. For some rules that really depend
 # on the identity of the process performing an action such as opening
@@ -42,6 +39,13 @@
 - macro: bin_dir
   condition: fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
 
+- macro: bin_dir_resolved
+  condition: >
+    (evt.abspath startswith /bin/ or
+     evt.abspath startswith /sbin/ or
+     evt.abspath startswith /usr/bin/ or
+     evt.abspath startswith /usr/sbin/)
+
 - macro: bin_dir_mkdir
   condition: >
     (evt.arg[1] startswith /bin/ or
@@ -241,14 +245,18 @@
 # Network
 - macro: inbound
   condition: >
-    (((evt.type in (accept,listen) and evt.dir=<)) or
+    (((evt.type in (accept,listen) and evt.dir=<) or
+      (evt.type in (recvfrom,recvmsg) and evt.dir=< and
+       fd.l4proto != tcp and fd.connected=false and fd.name_changed=true)) and
      (fd.typechar = 4 or fd.typechar = 6) and
      (fd.ip != "0.0.0.0" and fd.net != "127.0.0.0/8") and
      (evt.rawres >= 0 or evt.res = EINPROGRESS))
 
 - macro: outbound
   condition: >
-    (((evt.type = connect and evt.dir=<)) or
+    (((evt.type = connect and evt.dir=<) or
+      (evt.type in (sendto,sendmsg) and evt.dir=< and
+       fd.l4proto != tcp and fd.connected=false and fd.name_changed=true)) and
      (fd.typechar = 4 or fd.typechar = 6) and
      (fd.ip != "0.0.0.0" and fd.net != "127.0.0.0/8") and
      (evt.rawres >= 0 or evt.res = EINPROGRESS))
@@ -257,7 +265,9 @@
 # for efficiency.
 - macro: inbound_outbound
   condition: >
-    (((evt.type in (accept,listen,connect) and evt.dir=<)) or
+    (((evt.type in (accept,listen,connect) and evt.dir=<) or
+      (evt.type in (recvfrom,recvmsg,sendto,sendmsg) and evt.dir=< and
+       fd.l4proto != tcp and fd.connected=false and fd.name_changed=true)) and
      (fd.typechar = 4 or fd.typechar = 6) and
      (fd.ip != "0.0.0.0" and fd.net != "127.0.0.0/8") and
      (evt.rawres >= 0 or evt.res = EINPROGRESS))
@@ -368,13 +378,6 @@
        proc.pcmdline startswith "node /root/.config/yarn" or
        proc.pcmdline startswith "node /opt/yarn/bin/yarn.js"))
 
-
-- macro: httpd_writing_ssl_conf
-  condition: >
-    (proc.pname=run-httpd and
-     (proc.cmdline startswith "sed -ri" or proc.cmdline startswith "sed -i") and
-     (fd.name startswith /etc/httpd/conf.d/ or fd.name startswith /etc/httpd/conf))
-
 - macro: parent_Xvfb_running_xkbcomp
   condition: (proc.pname=Xvfb and proc.cmdline startswith 'sh -c "/usr/bin/xkbcomp"')
 
@@ -790,7 +793,6 @@
     and not centrify_writing_krb
     and not cockpit_writing_conf
     and not ipsec_writing_conf
-    and not httpd_writing_ssl_conf
 
 - rule: Write below etc
   desc: an attempt to write to any file below /etc
@@ -930,12 +932,7 @@
   condition: (proc.aname[2]=redis-server and (proc.cmdline contains "redis-server.post-up.d" or proc.cmdline contains "redis-server.pre-up.d"))
 
 - macro: rabbitmq_running_scripts
-  condition: >
-    (proc.pname=beam.smp and
-    (proc.cmdline startswith "sh -c exec ps" or
-     proc.cmdline startswith "sh -c exec inet_gethost" or
-     proc.cmdline= "sh -s unix:cmd" or
-     proc.cmdline= "sh -c exec /bin/sh -s unix:cmd 2>&1"))
+  condition: (proc.pname=beam.smp and (proc.cmdline startswith "sh -c exec ps" or proc.cmdline startswith "sh -c exec inet_gethost"))
 
 - macro: rabbitmqctl_running_scripts
   condition: (proc.aname[2]=rabbitmqctl and proc.cmdline startswith "sh -c ")
@@ -957,7 +954,7 @@
 
 - rule: Modify binary dirs
   desc: an attempt to modify any file below a set of binary directories.
-  condition: (bin_dir_rename) and modify and not package_mgmt_procs and not exe_running_docker_save
+  condition: (bin_dir_rename or bin_dir_resolved) and modify and not package_mgmt_procs and not exe_running_docker_save
   output: >
     File below known binary directory renamed/removed (user=%user.name command=%proc.cmdline
     operation=%evt.type file=%fd.name %evt.args)
@@ -1347,35 +1344,33 @@
 - list: statsd_ports
   items: [8125]
 
+- list: mysql_ports
+  items: [3306]
+
 - list: ntp_ports
   items: [123]
 
-# Some applications will connect a udp socket to an address only to
-# test connectivity. Assuming the udp connect works, they will follow
-# up with a tcp connect that actually sends/receives data.
+# 0 is included in the list because some apps connect to an address
+# only to test connectivity.
 #
-# With that in mind, we listed a few commonly seen ports here to avoid
-# some false positives. In addition, we make the main rule opt-in, so
-# it's disabled by default.
-
-- list: test_connect_ports
-  items: [0, 9, 80, 3306]
-
-- macro: do_unexpected_udp_check
-  condition: (never_true)
-
+# mysql_ports is included becuase some versions of the mysql client
+# will attempt a connect using udp + port 3306 before connecting via
+# tcp + port 3306.
+#
+# 80 is included for the same reason as mysql_ports--some apps do a
+# connect using udp before doing a real connect using tcp.
 - list: expected_udp_ports
-  items: [53, openvpn_udp_ports, l2tp_udp_ports, statsd_ports, ntp_ports, test_connect_ports]
+  items: [0, 53, 80, openvpn_udp_ports, l2tp_udp_ports, statsd_ports, mysql_ports, ntp_ports]
 
 - macro: expected_udp_traffic
   condition: fd.port in (expected_udp_ports)
 
 - rule: Unexpected UDP Traffic
   desc: UDP traffic not on port 53 (DNS) or other commonly used ports
-  condition: (inbound_outbound) and do_unexpected_udp_check and fd.l4proto=udp and not expected_udp_traffic
+  condition: (inbound_outbound) and fd.l4proto=udp and not expected_udp_traffic
   output: >
     Unexpected UDP Traffic Seen
-    (user=%user.name command=%proc.cmdline connection=%fd.name proto=%fd.l4proto evt=%evt.type %evt.args)
+    (user=%user.name command=%proc.cmdline connection=%fd.name proto=%fd.l4proto)
   priority: NOTICE
   tags: [network]
 

test/falco_test.py

@@ -31,7 +31,6 @@ class FalcoTest(Test):
 
         self.json_output = self.params.get('json_output', '*', default=False)
         self.json_include_output_property = self.params.get('json_include_output_property', '*', default=True)
-        self.all_events = self.params.get('all_events', '*', default=False)
         self.priority = self.params.get('priority', '*', default='debug')
         self.rules_file = self.params.get('rules_file', '*', default=os.path.join(self.basedir, '../rules/falco_rules.yaml'))
 
@@ -366,9 +365,6 @@ class FalcoTest(Test):
         if self.run_duration:
             cmd += ' -M {}'.format(self.run_duration)
 
-        if self.all_events:
-            cmd += ' -A'
-
         self.falco_proc = process.SubProcess(cmd)
 
         res = self.falco_proc.run(timeout=180, sig=9)

test/falco_tests.yaml

@@ -128,7 +128,6 @@ trace_files: !mux
       - rules/single_rule.yaml
       - rules/double_rule.yaml
     trace_file: trace_files/cat_write.scap
-    all_events: True
 
   rules_directory:
     detect: True
@@ -139,7 +138,6 @@ trace_files: !mux
     rules_file:
       - rules/rules_dir
     trace_file: trace_files/cat_write.scap
-    all_events: True
 
   multiple_rules_suppress_info:
     detect: True
@@ -155,7 +153,6 @@ trace_files: !mux
       - rules/single_rule.yaml
       - rules/double_rule.yaml
     trace_file: trace_files/cat_write.scap
-    all_events: True
 
   multiple_rules_overriding:
     detect: False
@@ -702,7 +699,6 @@ trace_files: !mux
       - detect_madvise: 2
       - detect_open: 2
     trace_file: trace_files/syscall.scap
-    all_events: True
 
   catchall_order:
     detect: True

userspace/falco/falco.cpp

@@ -22,8 +22,6 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 #include <fstream>
 #include <set>
 #include <list>
-#include <vector>
-#include <algorithm>
 #include <string>
 #include <signal.h>
 #include <fcntl.h>
@@ -34,7 +32,6 @@ along with falco.  If not, see <http://www.gnu.org/licenses/>.
 #include <sinsp.h>
 
 #include "logger.h"
-#include "utils.h"
 
 #include "configuration.h"
 #include "falco_engine.h"
@@ -154,8 +151,7 @@ uint64_t do_inspect(falco_engine *engine,
 		    falco_outputs *outputs,
 		    sinsp* inspector,
 		    uint64_t duration_to_tot_ns,
-		    string &stats_filename,
-		    bool all_events)
+		    string &stats_filename)
 {
 	uint64_t num_evts = 0;
 	int32_t res;
@@ -222,7 +218,8 @@ uint64_t do_inspect(falco_engine *engine,
 			}
 		}
 
-		if(!ev->falco_consider() && !all_events)
+		if(!inspector->is_debug_enabled() &&
+			ev->get_category() & EC_INTERNAL)
 		{
 			continue;
 		}
@@ -244,47 +241,6 @@ uint64_t do_inspect(falco_engine *engine,
 	return num_evts;
 }
 
-static void print_all_ignored_events(sinsp *inspector)
-{
-	sinsp_evttables* einfo = inspector->get_event_info_tables();
-	const struct ppm_event_info* etable = einfo->m_event_info;
-	const struct ppm_syscall_desc* stable = einfo->m_syscall_info_table;
-
-	std::set<string> ignored_event_names;
-	for(uint32_t j = 0; j < PPM_EVENT_MAX; j++)
-	{
-		if(!sinsp::falco_consider_evtnum(j))
-		{
-			std::string name = etable[j].name;
-			// Ignore event names NA*
-			if(name.find("NA") != 0)
-			{
-				ignored_event_names.insert(name);
-			}
-		}
-	}
-
-	for(uint32_t j = 0; j < PPM_SC_MAX; j++)
-	{
-		if(!sinsp::falco_consider_syscallid(j))
-		{
-			std::string name = stable[j].name;
-			// Ignore event names NA*
-			if(name.find("NA") != 0)
-			{
-				ignored_event_names.insert(name);
-			}
-		}
-	}
-
-	printf("Ignored Event(s):");
-	for(auto it : ignored_event_names)
-	{
-		printf(" %s", it.c_str());
-	}
-	printf("\n");
-}
-
 //
 // ARGUMENT PARSING AND PROGRAM SETUP
 //
@@ -314,7 +270,6 @@ int falco_init(int argc, char **argv)
 	string output_format = "";
 	bool replace_container_info = false;
 	int duration_to_tot = 0;
-	bool print_ignored_events = false;
 
 	// Used for writing trace files
 	int duration_seconds = 0;
@@ -344,7 +299,6 @@ int falco_init(int argc, char **argv)
 		{"version", no_argument, 0, 0 },
 		{"validate", required_argument, 0, 'V' },
 		{"writefile", required_argument, 0, 'w' },
-		{"ignored-events", no_argument, 0, 'i'},
 
 		{0, 0, 0, 0}
 	};
@@ -361,7 +315,7 @@ int falco_init(int argc, char **argv)
 		// Parse the args
 		//
 		while((op = getopt_long(argc, argv,
-                                        "hc:AdD:e:ik:K:Ll:m:M:o:P:p:r:s:T:t:UvV:w:",
+                                        "hc:AdD:e:k:K:Ll:m:M:o:P:p:r:s:T:t:UvV:w:",
                                         long_options, &long_index)) != -1)
 		{
 			switch(op)
@@ -387,9 +341,6 @@ int falco_init(int argc, char **argv)
 				k8s_api = new string();
 				mesos_api = new string();
 				break;
-			case 'i':
-				print_ignored_events = true;
-				break;
 			case 'k':
 				k8s_api = new string(optarg);
 				break;
@@ -480,20 +431,12 @@ int falco_init(int argc, char **argv)
 			return EXIT_SUCCESS;
 		}
 
+
 		inspector = new sinsp();
-
-		if(print_ignored_events)
-		{
-			print_all_ignored_events(inspector);
-			delete(inspector);
-			return EXIT_SUCCESS;
-		}
-
 		engine = new falco_engine();
 		engine->set_inspector(inspector);
 		engine->set_extra(output_format, replace_container_info);
 
-
 		outputs = new falco_outputs();
 		outputs->set_inspector(inspector);
 
@@ -818,8 +761,7 @@ int falco_init(int argc, char **argv)
 				      outputs,
 				      inspector,
 				      uint64_t(duration_to_tot*ONE_SECOND_IN_NS),
-				      stats_filename,
-				      all_events);
+				      stats_filename);
 
 		duration = ((double)clock()) / CLOCKS_PER_SEC - duration;